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ABSTRACT 


The  thesis  aims  to  discuss  and  analyze  the  establishment  of  a  methodical  and  repeatable 
method  to  develop  measures  of  effectiveness  (MOE)  for  a  condition-based  maintenance  plus 
(CBM-I-)  maintenance  management  program.  There  is  currently  no  consistent  method  in  use 
in  the  surface  maintenance  community  to  develop  CBM-i-  related  MOEs. 

A  set  of  eight  MOEs  is  developed  using  the  nine-step  method,  a  systems  engineering 
(SE)  focused  approach  that  uses  an  integrative  framework  of  objective  values  and  subjective 
criteria  to  guide  the  development  of  MOEs. 

A  maintenance  organization  may  use  the  developed  set  of  eight  CMMS  MOEs  to 
determine  how  well  the  CMMS  is  being  employed  within  the  organization  and  how  well  it 
supports  a  CBM-i-  approach.  The  MOE  set  provides  basic  indicators  to  determine  the 
effectiveness  of  the  CMMS  functions  and  of  the  system  as  an  object.  The  developed  MOEs 
also  address  the  initial  set  of  stakeholder  requirements  and  needs. 

Maintenance  organizations  can  use  the  development  processes  established  by  the 
nine-step  methodology  to  develop  valid,  significant,  and  useful  MOEs  for  system’s  fitness- 
for-purpose  evaluation  and  determination.  The  nine-step  method  is  practicable  within  a 
typical  surface  maintenance  organization. 
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EXECUTIVE  SUMMARY 


In  May  2013,  the  Naval  Sea  Systems  Command  (NAVSEA)  established  policy 
implementing  condition-based  maintenance  plus  (CBM-i-)  as  the  maintenance  approach  of 
choice  to  reduce  and  control  ship’s  lifecycle  costs  (Naval  Sea  Systems  Command  2013, 1). 
CBM-i-  implementation  represents  a  fundamental  shift  from  the  traditional  Navy  maintenance 
management  methods  used  to  control  preventive  maintenance  (PM)  and  corrective 
maintenance  (CM).  The  Navy’s  legacy  fleet  has  been  built  around  a  PM  and  CM  system  with 
a  foundation  on  reliability-centered  maintenance  (RCM)  analysis.  Platforms  that  support  a 
CBM-t  strategy  require  a  significant  quantity  of  sensors  and  meters  to  build  an  accurate 
maintenance  history,  as  well  as  a  computerized  maintenance  management  system  (CMMS) 
for  real  time  data  collection  and  display.  A  successful  CBM-i-  program  relies  on  the 
collection  and  management  of  accurate  real-time  system  component  data  that  enables 
decision  makers  to  increase  system  availability  by  scheduling  maintenance  only  when  it  is 
required.  The  correct  implementation  of  CBM-i-  should  result  in  reduced  total  ownership 
costs  and  improved  system  availability  (Department  of  Defense  2008,  1-3). 

The  Littoral  Combat  Ship  (LCS)  is  the  only  class  of  currently  deployed  ships 
implementing  the  CBM+  maintenance  approach  early  in  their  lifecycle.  The  LCS  program’s 
operational  concept  requires  contractors  to  manage  an  increasing  portion  of  the  ship’s 
maintenance  using  Commercial  Off-The-Shelf  (COTS)  CMMS  programs.  The  LCS’  non- 
traditional  reduced  manning  concept  requires  LCS  maintenance  managers  to  develop 
strategies  using  RCM  and  untried  condition-based  maintenance  (CBM)  approaches 
(Commander,  U.S.  Lleet  Lorces  Command  2013).  LCS  program  contractors  use  Maximo,  an 
IBM  developed  enterprise  asset  management  (EAM)  program,  to  manage  maintenance. 

At  their  core,  CMMS  programs  are  information  databases  with  customizable 
interfaces  to  aid  decision  makers  in  determining  the  best  path  to  manage  maintenance. 
Contractor-driven  maintenance  management  using  a  COTS  CMMS  program  is  a  sharp 
departure  from  the  traditional  surface  Navy  maintenance  management  process.  Successful 
CMMS  implementation  is  influential  a  CBM+  program’s  success. 


CMMS  programs  are  instrumental  in  the  effective  implementation  of  a  CBM+ 
strategy,  yet  there  is  a  lack  of  consistent  methodology  to  measure  CMMS  implementation 
effectiveness  as  it  pertains  to  CBM+.  The  problem  that  results  from  the  lack  of  a  well- 
defined  and  methodically  derived  set  of  measures  of  effectiveness  (MOE)  is  that  CBM-i- 
maintainers  cannot  properly  determine  or  monitor  how  well  the  selected  CMMS  program 
accomplishes  its  intended  operational  objectives.  MOEs  are  important  during  system 
selection  and  become  crucial  during  system  effectiveness  monitoring  and  change 
implementation  throughout  the  system’s  lifecycle  because  when  correctly  selected  they 
indicate  how  well  a  system  is  fit  for  its  intended  purpose.  Currently  there  is  no  method  that 
provides  a  set  of  objective  values  and  subjective  criteria  for  MOE  development. 

The  objective  of  this  thesis  is  to  utilize  a  consistent  systems  engineering  (SE) 
approach  to  develop  an  initial  set  of  MOEs  that  help  define  the  successful  operational  use  of 
a  CMMS  program  in  the  implementation  of  a  CBM-i-  strategy.  MOEs  will  be  developed 
using  a  framework  that  provides  a  consistent  methodical  approach  to  determine  the 
effectiveness  of  a  selected  CMMS. 

The  results  of  this  thesis  will  provide  an  initial  set  of  MOEs  for  the  usage  of  a  CMMS 
in  support  of  ship  maintenance.  The  MOEs  will  be  developed  using  a  consistent  analytical 
framework  method  that  allows  for  iterations.  The  CMMS  is  a  singular  component  of  a  more 
complex  system  required  for  the  successful  establishment  of  a  CBM-i-  strategy.  Although  it  is 
but  a  component  in  a  management  system,  it  often  contains  the  information  dashboard  used 
by  decision  makers  when  controlling  maintenance.  A  well-developed  set  of  CMMS  MOEs 
will  benefit  the  maintenance  community  in  determining  how  well  the  employed  system  is 
meeting  the  organization’s  needs. 

MOEs  were  characterized  using  a  modified  application  of  Langford’s  nine-step 

method  that  allows  the  MOE  development  concept  to  become  repeatable,  and  able  to  be 

validated  with  operational  and  managerial  needs  (G.  Langford,  Determinants  of  Deterrence 

Effectiveness  2014,  7-12).  This  research  only  used  the  first  seven  steps  of  the  nine-step 

method.  The  last  two  steps  are  better  used  during  MOE  reevaluation.  The  seven  steps  used  in 

this  study  and  their  application  to  CMMS  MOE  development  are  described  below  (G. 

Langford,  Determinants  of  Deterrence  Effectiveness  2014,  7-12): 

xviii 


•  Define  terminology:  Defined  CMMS,  CBM+,  and  other  maintenance 
management  working  terms  to  determine  the  research  scope. 

•  Delineate  boundaries  and  functions:  Performed  a  functional  analysis  of  the 
key  measurable  CMMS  and  management  functions. 

•  Perform  lifecycle  analysis:  Defined  the  life  cycle  of  the  key  measurable 
CMMS  and  management  functions. 

•  Define  requirements :  Derived  stakeholder  requirement  by  analyzing  solicited 
stakeholder  needs. 

•  Postulate  solution  set:  Developed  a  set  of  solutions  to  satisfy  the  problem 
domain  issues. 

•  Determine  theoretical  foundations:  Applied  management  maintenance  and 
logistics  theory  to  the  problem  issue. 

•  Formalize  framework:  Mapped  subjective  elements  to  objective  elements 
using  the  integrative  framework. 

The  following  is  a  summary  of  the  CMMS  MOEs  were  developed  using  the  integrative 
framework. 


MOE 

Description 

MOE-e:  CMMS 
experience 

Cognition-Object:  Pro\ides  insight  into  the  effectiveness  of 
system  usage  as  it  relates  to  recording,  updating, 
coordination,  and  communication  of  maintenance  data. 

MOE-r:  CMMS 

Response 

Cognition-Object:  Pro\ides  insight  into  the  effectiveness  of 
the  management  and  repotting  of  maintenance  data  and  the 
system’s  interface  w’ith  objects. 

MOE-g:  CMMS 

Functionality 

Prediction 

Cognition-Function:  Pro\ides  insight  into  the  effectiveness 
of  the  predicted  functional  results  arising  from  interactions 
between  the  systems  and  data  repotting  objects. 

MOE-c:  CMMS 

Functionality 

Expectation 

Cognition-Function:  Identifies  user  satisfactionbasedon 
system  expectations. 

MOE-s:CMMS 
Selection  Validity 

Procedure-Object:  Identifies  the  high  level  stakeholder’s 
satisfaction  based  on  programmatic  goals. 

MOE-x:CMMS 

Operational 

Context  Validity 

Procedure-Object:  Identifies  the  CMMS  ’  ability  to  interact 
with  external  objects. 

MOE-u:CMMS 

Functional 

Resource 

Utilization  V  alidity 

Procedure-Function:  Pro\ides  insight  into  the  efficiency  of 
the  CMMS’  functions  to  support  resource  management. 

MOE-b:CMMS 

Functional 

Boundary 

Conditions 

Procedure-Function:  Pro\ides  a  metric  of  the  CMMS  ’ 
function’s  ability  to  interact  with  external  objects. 

Figure  1  CMMS  MOE  Summary 
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A  maintenance  organization  may  use  the  developed  set  of  eight  CMMS  MOEs  to 
determine  how  well  the  CMMS  is  being  employed  within  the  organization  and  how  well  it 
supports  a  CBM+  approach.  The  MOE  set  provides  basic  indicators  to  determine  the 
effectiveness  of  the  CMMS  function’s  and  of  the  system  as  an  object.  The  developed  MOEs 
also  address  the  initial  set  of  stakeholder  requirements  and  needs. 

Maintenance  organizations  can  use  the  development  processes  established  by  the 
nine-step  methodology  to  develop  valid,  significant,  and  useful  MOEs  for  system’s  fitness- 
for-purpose  evaluation  and  determination.  The  nine-step  method  is  practicable  within  a 
typical  surface  maintenance  organization.  Developing  and  iterating  MOEs  will  only  require 
staff  with  an  understanding  of  the  overarching  programmatic  goals  and  knowledge  of  the 
evaluated  systems  due  to  the  method’s  simple  and  direct  approach.  The  process  could  also  be 
extended  to  MOE  development  in  other  areas. 
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I.  INTRODUCTION 


A.  A  FUNDAMENTAL  SHIFT  IN  MAINTENANCE  MANAGEMENT 

In  May  2013,  the  Naval  Sea  Systems  Command  (NAVSEA)  established  policy 
implementing  condition-based  maintenance  plus  (CBM-i-)  as  the  maintenance  approach  of 
choice  to  reduce  and  control  ship’s  lifecycle  costs  (Naval  Sea  Systems  Command  2013,  1). 
CBM-i-  implementation  represents  a  fundamental  shift  from  the  traditional  Navy  maintenance 
management  methods  used  to  control  preventive  maintenance  (PM)  and  corrective 
maintenance  (CM).  The  Navy’s  legacy  fleet  has  been  built  around  a  PM  and  CM  system  with 
a  foundation  on  reliability-centered  maintenance  (RCM)  analysis.  Platforms  that  support  a 
CBM-t  strategy  require  a  significant  quantity  of  sensors  and  meters  to  build  an  accurate 
maintenance  history,  as  well  as  a  computerized  maintenance  management  system  (CMMS) 
for  real  time  data  collection  and  display.  A  successful  CBM-i-  program  relies  on  the 
collection  and  management  of  accurate  real-time  system  component  data  that  enables 
decision  makers  to  increase  system  availability  by  scheduling  maintenance  only  when  it  is 
required.  The  correct  implementation  of  CBM-i-  should  result  in  reduced  total  ownership 
costs  and  improved  system  availability  (Department  of  Defense  2008,  1-3). 

The  Littoral  Combat  Ship  (LCS)  is  the  only  class  of  currently  deployed  ships 
implementing  the  CBM-i-  maintenance  approach  early  in  their  lifecycle.  The  LCS  program’s 
operational  concept  requires  contractors  to  manage  an  increasing  portion  of  the  ship’s 
maintenance  using  Commercial  Off-The-Shelf  (COTS)  CMMS  programs.  The  LCS’  non- 
traditional  reduced  manning  concept  requires  LCS  maintenance  managers  to  develop 
strategies  using  RCM  and  untried  condition-based  maintenance  (CBM)  approaches 
(Commander,  U.S.  Lleet  Lorces  Command  2013).  LCS  program  contractors  use  Maximo,  an 
IBM  developed  enterprise  asset  management  (EAM)  program,  to  manage  maintenance. 

At  their  core,  CMMS  programs  are  information  databases  with  customizable 
interfaces  to  aid  decision  makers  in  determining  the  best  path  to  manage  maintenance. 
Contractor-driven  maintenance  management  using  a  COTS  CMMS  program  is  a  sharp 
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departure  from  the  traditional  surface  Navy  maintenance  management  process.  Successful 
CMMS  implementation  is  influential  a  CBM+  program’s  success. 

B.  PROBLEM  STATEMENT 

CMMS  programs  are  instrumental  in  the  effective  implementation  of  a  CBM+ 
strategy,  yet  there  is  a  lack  of  consistent  methodology  to  measure  CMMS  implementation 
effectiveness  as  it  pertains  to  CBM+.  The  problem  that  results  from  the  lack  of  a  well- 
defined  and  methodically  derived  set  of  measures  of  effectiveness  (MOE)  is  that  CBM-i- 
maintainers  cannot  properly  determine  or  monitor  how  well  the  selected  CMMS  program 
accomplishes  its  intended  operational  objectives.  MOEs  are  important  during  system 
selection  and  become  crucial  during  system  effectiveness  monitoring  and  change 
implementation  throughout  the  system’s  lifecycle  because  when  correctly  selected  they 
indicate  how  well  a  system  is  fit  for  its  intended  purpose.  Currently  there  is  no  method  that 
provides  a  set  of  objective  values  and  subjective  criteria  for  MOE  development. 

C.  OBJECTIVE  AND  SCOPE 

The  objective  of  this  thesis  is  to  utilize  a  consistent  systems  engineering  (SE) 
approach  to  develop  an  initial  set  of  MOEs  that  help  define  the  successful  operational  use  of 
a  CMMS  program  in  the  implementation  of  a  CBM-i-  strategy.  MOEs  will  be  developed 
using  a  framework  that  provides  a  consistent  methodical  approach  to  determine  the 
effectiveness  of  a  selected  CMMS. 

The  scope  of  this  research  includes  the  development  of  an  initial  set  of  CMMS  MOEs 
based  on  user  needs  and  requirements  to  measure  the  success  of  the  operational  use  of  the 
CMMS.  Needs  and  requirements  are  derived  from  the  ECS  CONOPS,  established  policy 
requirements,  and  stakeholder  input.  This  thesis  uses  the  ECS  program  as  a  case  study  because 
they  are  the  surface  program  that  has  implemented  a  CBM-i-  approach  for  the  longest  time.  The 
ECS  program  has  also  implemented  the  CBM-i-  approach  early  in  their  lifecycle  as  opposed  to 
other  class  types  that  have  implemented  traditional  maintenance  approaches  for  decades. 

Eor  the  purpose  of  this  thesis  the  CMMS  program  strictly  refers  to  its  use  as  a 
maintenance  management  system  and  not  as  an  EAM  program.  Although  the  chosen  CMMS 
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might  have  the  capability  to  collect  and  analyze  the  required  data  to  function  as  an  EAM,  its 
use  in  the  maintenance  program  is  limited  to  maintenance  management.  EAM  functionalities 
require  the  use  of  a  CMMS  to  manage  funds,  personnel,  and  a  supply  chain  among  other 
items. 


1.  Research  Questions 

•  How  can  the  fitness-for-purpose  of  a  CMMS  program  be  measured? 

•  How  can  a  singular  reproducible  method  be  used  to  develop  CBM+ 
maintenance  management  MOEs? 

•  How  can  CMMS  MOEs  be  developed  using  both  objective  and  subjective 
criteria? 

D.  SIGNIFICANCE  OF  THE  STUDY 

The  results  of  this  thesis  will  provide  an  initial  set  of  MOEs  for  the  usage  of  a  CMMS 
in  support  of  ship  maintenance.  The  MOEs  will  be  developed  using  a  consistent  analytical 
framework  method  that  allows  for  iterations.  The  CMMS  is  a  singular  component  of  a  more 
complex  system  required  for  the  successful  establishment  of  a  CBM+  strategy.  Although  it  is 
but  a  component  in  a  management  system,  it  often  contains  the  information  dashboard  used 
by  decision  makers  when  controlling  maintenance.  A  well-developed  set  of  CMMS  MOEs 
will  benefit  the  maintenance  community  in  determining  how  well  the  employed  system  is 
meeting  the  organization’s  needs. 

E.  ORGANIZATION  OF  STUDY 

•  Chapter  I:  Introduction 

•  Chapter  II:  Navy  Ship  Maintenance  and  CMMS  Background 

•  Chapter  III:  Eiterature  Review:  Measures  of  Effectiveness 

•  Chapter  IV :  Methodology 

•  Chapter  V:  Measures  of  Effectiveness  Development 

•  Chapter  VI:  Measures  of  Effectiveness  Set  Discussion 
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II.  NAVY  SHIP  MAINTENANCE  AND  CMMS  BACKGROUND 


It  is  useful  to  understand  traditional  maintenance  strategies  to  fully  grasp  condition- 
based  maintenance  plus  (CBM-i-)  as  a  concept  and  how  it  benefits  from  the  correct 
implementation  of  a  computerized  maintenance  management  system  (CMMS)  program.  This 
chapter  provides  a  brief  overview  of  the  traditional  surface  Navy  maintenance  strategies.  It 
also  provides  a  brief  background  on  CMMS  as  a  system  and  its  selection  process. 
Understanding  the  process  of  justifying  and  selecting  the  CMMS  provides  insight  on  how  to 
better  approach  the  development  of  measures  of  effectiveness  (MOE). 

A.  MAINTENANCE  OVERVIEW 

The  International  Council  on  Systems  Engineering  (INCOSE)  (1998,  136-137) 
defines  maintenance  as  “those  actions  required  to  restore  or  maintain  an  item  to  a  serviceable 
condition”  and  “the  combination  of  all  technical  and  administrative  actions,  including 
supervision  actions,  intended  to  retain  an  item  in,  or  restore  it  to,  a  state  in  which  it  can 
perform  a  required  function. 

Organizations  may  use  several  maintenance  management  approaches  and  strategies 
based  on  their  maintenance  needs.  Traditionally  all  maintenance  strategies  seek  to  keep  or 
upkeep  an  object  in  a  materiel  condition  in  which  it  can  operate  as  designed. 

I.  Surface  Navy  Maintenance 

Navy  ship  maintenance  is  categorized  as  proactive  or  corrective  maintenance. 

a.  Proactive  Maintenance 

Navy  proactive  maintenance  is  accomplished  using  three  approaches: 

•  reliability-centered  maintenance  (RCM) 

•  preventive  maintenance  (PM) 

•  CBM-i- 
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(1)  Reliability-Centered  Maintenance 

RCM  is  a  logical  function-oriented  process  that  identifies  optimal  equipment 
maintenance  requirements  based  on  reliability  characteristics  and  operating  conditions.  Its 
goal  is  to  eliminate  PM  tasks  that  do  not  add  value  to  the  maintenance  process  by  using 
reliability  data  to  adjust  maintenance  time  intervals.  John  Moubray  (1997,  28)  defines  RCM 
as  “a  process  used  to  determine  the  maintenance  requirements  of  any  physical  asset  in  its 
operating  context”.  This  method  was  developed  in  the  1970s  as  a  response  to  the  traditional 
approach  of  scheduled  maintenance  that  assumed  that  every  system  has  a  specific  periodicity 
at  which  maintenance  is  required  to  prevent  failure  (Moubray  1997, 2-6).  The  application  of 
RCM  methods  produces  analysis  that  is  useful  in  the  development  of  a  cost-effective  PM 
program’s  requirements  (Blanchard  and  Fabrycky,  Systems  Engineering  and  Analysis  2011, 
439).  RCM  seeks  to  answer  seven  basic  maintenance  questions  (Moubray  1997,  7): 

•  What  are  the  functions  and  associated  performance  standards  of  the  asset  in 
its  present  operating  context? 

•  In  what  way  does  it  fail  to  fulfil  its  functions? 

•  What  causes  each  functional  failure? 

•  What  happens  when  each  failure  occurs? 

•  In  what  way  does  each  failure  matter? 

•  What  can  be  done  to  predict  or  prevent  each  failure? 

•  What  should  be  done  if  a  suitable  proactive  task  cannot  be  found? 

RCM  takes  advantage  of  the  predictive  properties  of  equipment  performance  by 
using  condition  discovery  methods  such  as  scheduled  inspections  and  rotating  equipment 
vibration  analysis.  The  RCM  data  produced  during  maintenance  tasks  is  analyzed  and  used 
to  identify  or  modify  required  task  periodicity  and  execution.  A  well-implemented  RCM 
strategy  achieves  greater  equipment  safety,  performance,  maintenance  cost  effectiveness, 
useful  life,  and  better  equipment  database  (Moubray  1997,  18-20). 

RCM  methods  serve  as  the  foundation  for  PM  and  condition-based  maintenance 
(CBM)  strategies. 
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Figure  1  displays  a  RCM  decision  diagram  adapted  from  the  work  of  Nowlan  and 
Heap  (1978).  The  decision  tree  shows  a  series  of  questions  that  should  be  asked  to  determine 
the  consequences  of  an  item’s  functional  failure  and  the  required  PM  to  prevent  the  failure. 
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Figure  1. 


RCM  decision  diagram  (after  Nowlan  and  Heap  1978,  92-93) 
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(2)  Preventive  Maintenance 

PM  is  a  time-driven  maintenance  strategy  based  on  calendar  time,  equipment 
operating  time,  or  a  number  of  cycles  (Department  of  Defense  2008,  2-2).  PM  is  intended  to 
prevent  unscheduled  downtime  or  damage  that  may  lead  to  a  catastrophic  failure.  Surface 
Navy  PM  typically  includes  all  scheduled  maintenance  actions  that  are  performed  to  ensure  a 
system  is  maintained  in  satisfactory  operational  materiel  condition.  PM  actions  and 
periodicity  schedules  are  generated  by  system  manufacturers  and  are  adjusted  throughout  the 
system’s  lifecycle  based  on  maintainer  feedback  and  reliability  analysis  results. 

There  are  three  levels  of  PM.  Onboard  a  legacy  ship,  organizational  level  PM  actions 
are  scheduled,  planned,  and  performed  by  ship’s  force  (SF).  In  the  LCS  program  contractors 
are  tasked  with  executing  the  organizational  level  PM  actions.  Intermediate  and  depot  level 
PM  actions  are  scheduled  and  planned  by  the  ship’s  maintenance  team  which  includes  the 
Port  Engineer  (PE),  the  ship’s  commanding  officer  who  is  represented  by  the  ship’s 
maintenance  and  materiel  officer  (SMMO),  and  a  representative  of  the  activity  executing  the 
maintenance. 

The  Navy  program  of  record  used  by  ship’s  force  in  both  LCS  and  legacy  ships  is  a 
CMMS  called  SKED.  SKED  has  been  the  surface  Navy’s  CMMS  since  2004.  It  creates, 
manages,  and  documents  PM  schedules  for  equipment  (Antech  Systems  2013). 

PM  strategy  advantages  include  (Mobley  2002,  416): 

•  Management:  PM  allows  managers  to  conveniently  schedule  personnel  and 
equipment  workload  around  the  maintenance  action. 

•  Parts:  PM  requires  a  smaller  quantity  of  replacement  parts  are  required  in 
stock  because  the  maintainer  can  predict  the  items  required  over  a  time 
period. 

•  Availability:  A  successful  PM  program  keeps  equipment  operation  within 
tolerance  as  it  requires  the  system  to  be  constantly  maintained  to  a  standard. 

PM  strategy  disadvantages  include  (Mobley  2002,  417-418): 

•  Potential  to  do  damage:  PM  may  require  performing  unnecessary 
maintenance  actions  on  equipment.  There  exists  the  potential  for  damage  to 
be  inflicted  any  time  a  system  is  touched  for  maintenance. 
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•  Infant  mortality:  New  replacement  parts  have  a  higher  probability  of 
failure. 

•  Parts:  PM  will  result  in  perfectly  functional  parts  to  be  replaced  and 
discarded. 

These  PM  advantages  and  disadvantages  lists  are  not  exhaustive  but  contain  six  of 
the  most  common  characteristics  that  describe  the  PM  approach. 

(3)  Condition-Based  Maintenance  Plus 

CBM-i-  is  a  maintenance  strategy  derived  from  RCM.  CBM-i-  uses  real  time  or  near 
real  time  system  reliability  sensor  data,  and  other  RCM  methods  to  determine  the  best 
maintenance  requirements  for  a  system.  According  to  the  Defense  Acquisition  Guidebook 
(DAG),  CBM-t: 

Can  be  useful  in  cost  effectively  sustaining  performance.  It  is  the  application 
and  integration  of  appropriate  processes,  technologies,  and  knowledge-based 
capabilities  to  improve  the  reliability  and  maintenance  effectiveness  of 
Department  of  Defense  (DOD)  systems  and  components.  At  its  core,  CBM-i- 
is  maintenance  performed  based  on  evidence  of  need  provided  by  RCM 
analysis  and  other  enabling  processes  and  technologies.  CBM-i-  uses  a 
systems  engineering  approach  to  collect  data,  enable  analysis,  and  support  the 
decision-making  processes  for  system  acquisition,  sustainment,  and 
operations.  (Department  of  Defense  2013,  414) 

The  following  is  a  selected  list  of  goals  resulting  from  successful  CBM-i- 
implementation.  Each  selected  goal  has  an  italicized  portion  that  specifies  the  importance  of 
an  effective  CMMS  to  manage  equipment  within  a  CBM-i-  approach: 

•  Enhance  maintenance  efficiency  and  effectiveness  and  integrate  all 
functional  aspects  of  life  cycle  management  processes  for  materiel 
requirements,  such  as  systems  engineering,  development,  acquisition, 
distribution,  supply  chain  management,  sustainment,  and 
modemization.(Department  of  Defense  2012,  5) 

•  Establish  integrated,  predictive  maintenance  approaches,  which  minimize 
unscheduled  repairs,  eliminate  unnecessary  maintenance,  and  employ  the 
most  cost-effective  system  health  management  processes.  (Department  of 
Defense  2012,  5) 

•  Implement  data  collection  and  analysis  requirements ...  to  measure  equipment 
sustainment  performance  characteristics  and  supporting  measures  of 
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effectiveness  throughout  life  cycle  sustainment.  (Department  of  Defense 
2012,  5) 

•  Improve  materiel  reliability  through  the  disciplined  analysis  of  failure  data  to 
modify  designs  and  operating  practices  to  ensure  equipment  meets  target 
performance  standards  within  operational  context.  (Department  of  Defense 
2012,  5) 

•  Optimize  life  cycle  logistics  processes  and  minimize  mean  downtime  by 
providing  timely  condition  information,  precise  failure  mode  identification, 
and  accurate  technical  data  that  will  expedite  repair  and  support  processes. 
(Department  of  Defense  2012,  5) 

Figure  2  is  an  illustrated  representation  of  the  CBM+  concept  and  its  relationship 
with  CBM,  RCM,  and  other  supporting  items. 


Functionality 


Hardware/ 

Software 


Tools 


Design 


Communications 


Figure  2.  CBM+  overview  (from  Department  of  Defense  2008,  1-4) 


In  2008,  DOD  policy  established  CBM-i-  as  the  primary  reliability  driver  in  the  total 
life  cycle  systems  management  strategy.  CBM-i-  is  designed  to  meet  the  following  DOD 
needs  (Department  of  Defense  2008,  3-7): 

•  Equipment  failure  prediction 

•  Equipment  condition  holistic  view 

•  Greater  accuracy  in  failure  prediction 
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•  Reduce  the  cost  of  ownership 

•  Improve  equipment  and  component  reliability 

•  Reduce  equipment  mean  down  time  (logistics  responsiveness) 

•  Optimize  equipment  performance  (availability) 

CBM+  implementation  requires  embedded  sensors  for  equipment  monitoring  and 
data  transmission.  CBM+  implementation  early  in  a  ship’s  lifecycle  provides  early  visibility 
into  equipment  performance  trends  that  may  result  in  failures.  The  equipment  performance 
history  developed  from  an  early  implementation  could  potentially  lead  to  improving 
maintenance  and  lower  maintenance  costs.  Since  the  goal  of  CBM+  is  to  maintain  based  on 
evidence,  the  issue  of  what  constitutes  evidence  may  involve  more  than  a  sensor  output. 
Other  factors  include  the  analysis  of  the  equipment  data  history  and  how  well  the  data  is 
managed. 

b.  Corrective  Maintenance 

Corrective  maintenance  (CM)  is  reactive  maintenance.  It  includes  all  unscheduled 
maintenance  actions  executed  as  a  result  of  system  malfunction,  failure,  or  deterioration 
(Navy  Personnel  Command  2013).  CM  scheduling  and  planning  urgency  is  determined  by 
the  severity  and  emergence  of  the  casualty  and  its  impact  on  the  ship’s  mission.  The 
emergent  nature  of  CM  work  usually  results  in  high  labor  costs  and  in  unexpected  system 
downtime.  Depending  on  a  system’s  complexity,  the  amount  of  time  spent  isolating,  finding, 
troubleshooting,  and  correcting  a  failure  during  CM  can  range  from  minutes  to  days  and 
even  weeks.  Figure  3  displays  a  general  flowchart  of  the  CM  cycle. 
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Detection 


Failure  Confirmed 
(Notification) 


Active  Maintenance  Commences 


Faulty  Item  Identified 


Disassembly  Completed 


Reassembly  Completed 


Repair  Completed  — 


Figure  3.  Corrective  maintenance  cycle  (from  Blanchard  2004,  59) 


2.  Maintenance  Overview  Summary 

Figure  4  is  a  graphical  summary  of  the  DOD  maintenance  approaches  used  in  the 
Navy.  The  two  overarching  maintenance  categories  are  proactive  and  reactive.  Proactive 
maintenance  splits  into  preventive  and  predictive  strategies,  both  of  which  are  based  on 
RCM. 
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Maintenance  Approaches 


Reactive 

Proactive 

Category 

Run-to-fail 

Preventive 

Predictive 

Sub-Category 

Fix  when  It  breaks 

Scheduled 

maintenance 

Condition-based 

maint.-dlagnostic 

Condition-based 
maim.-  prognostic 

When 

Scheduled 

No  scheduled 
maintenance 

Maintenance  based 
on  a  fixed  time  schedule 
for  inspect,  repair  and 
overhaul 

Maintenance  based 
on  current 
condition 

Maintenance  based 
on  forecast  of 
remaining  equipment 
life 

Why  Scheduled 

N/A 

Intolerable  failure  effect 
and  it  is  possible  to 
prevent  the  failure  effect 
through  a  scheduled 
overhaul  or  replacement 

Maintenance 
scheduled  based  on 
evidence  of  need 

Maintenance  need  is 
projected  as 
probable  within 
mission  time 

How  Scheduled 

N/A 

Based  on  the  useful  life 
of  the  component 
forecasted  during 
design  and  updated 
through  experience 

Continuous  collection 
of  condition 
monitoring  data 

Forecasting  of 
remaining  equipment 
life  based  on  actual 
stress  loading 

Kind  of 
Prediction 

None 

None 

On-  and  off-system, 
near-real-time  trend 
analysis 

On-  and  off-system, 
real-time  trend 
analysis 

Figure  4.  Range  of  maintenance  approaches  (from  Department  of  Defense 

2008,  2-3) 

B.  COMPUTERIZED  MAINTENANCE  MANAGEMENT  SYSTEMS 
I.  Overview 

CMMS  programs  are  fundamental  database  tools  that  assist  maintenance  activities  in 
decision  making  regarding  planning,  management,  and  administration  of  maintenance 
actions  required  in  a  CBM+  strategy.  The  CMMS  is  an  interactive  equipment  parameter  and 
condition  database  that  can  receive  data  input  from  users  or  from  sensors  and  meters.  The 
equipment  data  is  stored  as  a  system-specific  maintenance  history  that  may  be  used  by 
decision  makers  to  detect  trends  and  perform  analysis. 


CMMS  programs  typically  interface  with  users  through  customizable  modules 
chosen  based  on  the  maintenance  data  requirements.  Some  of  the  most  common  CMMS 
modules  perform  data  analysis  related  to  (Wireman  2009,  v): 


•  Preventive  maintenance 

•  Parts  inventory 

•  Parts  procurement 
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•  Work-order  management 

•  Labor  costs 

•  Material  costs 

•  Contracting  cost 

Figure  5  is  an  illustration  of  the  traditional  core  CMMS  activities.  The  illustration 
displays  the  division  of  CMMS  activities  into  four  main  activities  that  split  into  18  sub¬ 
activities. 


Figure  5.  Typical  CMMS  activities  (from  Raouf,  Ali  and  Duffuaa  1993,  39) 

Over  the  last  20  years  CMMS  software  has  continuously  improved  to  provide  better 
functionality  to  the  user.  The  CMMS  aspects  that  have  most  evolved  are  related  to 
application  integration,  planning  and  scheduling  capabilities,  analytical  tools  in  support  of 
RCM  and  failure  analysis,  and  integration  with  mobile  technology  (Berger  2011).  The 
effective  implementation  and  use  of  CMMS  software  may  result  in  increased  labor 
productivity,  increased  equipment  availability,  longer  equipment  useful  life,  reduced 
inventory  costs,  and  an  increase  in  overall  system  productivity  (Bagadia  2006,  1 1 1- 1 12).  A 
CMMS  may  help  accomplish  this  by  collecting  and  displaying  essential  maintenance 
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information  to  assist  decision  makers  to  better  manage  the  trends  and  requirements  of  a 
CBM+  maintenance  approach. 

Many  corporations  that  utilize  a  CMMS  as  part  of  their  maintenance  program  fail  to 
use  the  system  correctly  resulting  in  wasted  funds,  time,  and  effort.  A  2005  survey  shows 
that  94%  of  300  companies  fail  to  reap  the  full  benefits  of  their  CMMS  program  (Bagadia 
2006,  234).  The  most  common  reasons  for  unsuccessful  CMMS  implementation  include: 

•  Limiting  the  CMMS  to  act  as  a  database  instead  of  an  analysis  tool  (Arnold 
2006) 

•  Implementing  unnecessary  and  unmanageable  modules  (Arnold  2006) 

•  Sacrificing  training  for  time  and  cost  savings  (Arnold  2006)  (Bagadia  2006, 
235) 

•  Wrong  CMMS  selection  (Bagadia  2006,  196) 

•  Failure  to  populate  the  CMMS  with  system  data  such  as  parts  inventory 
(Arnold  2006)  (Wireman  2009,  129-130) 

•  Poor  interface  with  other  management  software  (Arnold  2006)  (Wireman 
2009,  128) 

•  Failure  to  develop  a  CMMS  implementation  plan  (Hemming  and  Davis  2003) 

Many  corporations  mistakenly  approach  CMMS  technology  as  a  ‘simple  and  quick 
fix’  by  failing  to  realize  that  the  successful  implementation  of  the  maintenance  system  relies 
on  process  discipline,  support  systems,  and  organizational  capabilities  (Autin  1998,  33-35). 
“Top  organizations  start  by  understanding  their  maintenance  environment.  They  analize  their 
business  needs,  match  a  system  and  processes  to  those  needs,  and  apply  formal  processes  to 
select  and  implement  the  new  system”  (Davis  and  Mikes  2006).  Succesful  implementation 
and  utilization  of  a  CMMS  is  preceded  by  the  performance  of  a  justification  analysis,  a 
thorough  system  selection  process,  and  a  systematic  implementation  process  (Wireman 
2009,  3-11). 


a.  Justification  Analysis 

A  study  to  justify  the  necessity  of  a  CMMS  requires  an  identification  of  needs  based 
on  current  practices,  an  evaluation  of  effectiveness  within  the  organization’s  maintenance 
process  and  identification  of  improvement  opportunities  (Bagadia  2006,  103-105). 
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In  2000,  the  Plant  Maintenance  Resource  Center  polled  77  CMMS  using  companies 
about  the  factors  considered  in  choosing  a  CMMS.  Table  1  lists  the  top  answers  provided. 
According  to  the  survey,  only  25  companies  chose  a  CMMS  to  meet  a  need  to  improve 
maintenance  performance. 


Table  1.  Factors  considered  in  justifying  CMMS  implementation  (from  The 

Plant  Maintenance  Resource  Center  2009) 


What  was  the  main  reason  that  your  workplace  changed  CMMS,  or 

implemented  £ 

new  CMMS? 

Reason 

Responses 

%  of  Total 

To  improve  maintenance  performance 

25 

28.7% 

Improved  functionality  and  features 

16 

18  4% 

To  integrate  the  Maintenance  system  with  other  systems 

9 

10.3% 

Don't  know 

8 

92% 

Year  2000  compliance  problems 

7 

8  0% 

Vendor  no  longer  supported  our  old  CMMS 

2 

2.3% 

To  comply  with  company  standards 

2 

2.3% 

To  use  newer  technology 

1 

11% 

Other 

7 

8.0% 

The  reasons  provided  by  these  corporations  to  justify  the  implementation  of  a  new 
CMMS  show  the  variety  of  organizational  needs  the  program  is  required  to  satisfy.  A 
company  that  justifies  new  CMMS  implementation  based  on  improving  maintenance 
performance  should  measure  implementation  effectiveness  using  different  factors  than  one 
that  chooses  a  new  CMMS  to  integrate  different  systems.  There  is  no  standard  set  of  MOEs 
that  will  capture  a  CMMS  implementation’s  fitness-for-purpose  for  all  different  users. 

b.  System  Selection 

Proper  CMMS  software  selection  is  instrumental  for  the  successful  implementation 
and  use  of  the  tool.  The  correct  implementation  of  COTS  CMMS  software  requires  a  needs 
analysis  to  ensure  that  the  chosen  program  and  its  implementation  achieve  the  intended 
mission.  The  first  step  in  the  selection  process  is  to  ensure  that  the  company’s  needs,  scope, 
goals,  and  objectives,  as  they  relate  to  CMMS,  are  well  defined  (Bagadia  2006,  121-133). 
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This  step  helps  prevent  selecting  a  CMMS  based  on  the  desire  to  obtain  a  “silver  bullet  to 
magically  fix  inefficiency  and  ineffectiveness”  (Autin  1998,  33). 

The  following  are  some  of  the  most  commonly  evaluated  criteria  while  selecting  a 
CMMS: 

•  Ease  of  learning  -  Software  should  include  training  modules  and  instructions 
documentation.  (Bagadia  2006,  129) 

•  Ease  to  use  -  The  software  should  be  menu  driven,  with  customizable 
screens,  error  handling  help,  and  useable  by  maintainers  with  no 
programming  experience.  (Bagadia  2006,  129) 

•  System  limitations  -  Particularly  the  system’s  ability  to  properly  process  and 
analyze  the  required  amount  of  data  without  significant  degradation  in 
performance.  (Bagadia  2006,  127) 

•  Software  flexibility  -  The  software  should  allow  for  expansion  and 
modification  based  on  user  needs.  (Bagadia  2006,  127) 

•  Software  compatibility  -  The  software  should  be  compatible  with  the  user’s 
operating  system.  (Bagadia  2006,  127)  (Caviedes  and  Knecht  2006) 
(Ouellette  2005) 

•  Interface  capabilities  -  The  software  should  interface  with  other  maintenance 
management  system  the  user  employs.  (Bagadia  2006,  127)  (Caviedes  and 
Knecht  2006) 

•  Security  -  Software  should  provide  a  backup  facility  to  prevent  any  data  loss. 
(Bagadia  2006,  128) 

•  Vendor  stability  and  technical  support  for  implementation  -  The  software 
vendor  should  have  proven  field  longevity  and  provide  technical  support  in 
areas  to  include:  update  releases,  engineering  consulting,  data  collection,  data 
entry,  training,  and  general  implementation.  (Ouellette  2005)  (Bagadia  2006, 
130) 

•  System  cost  -  To  include  cost  of  core  program  and  expandable  modules. 
(Ouellette  2005) 

c.  CMMS  Implementation 

CMMS  implementation  involves  a  series  of  steps  that  include  installing  and 
configuring  the  software,  fully  defining,  reviewing,  and  implementing  “all  workflow 
processes,  data-recording  requirements,  management  reports,  and  performance  metrics” 
(Bertolini  2009).  Figure  6  is  a  flowchart  for  a  typical  CMMS  implementation  process. 


18 


Figure  6.  CMMS  implementation  flowchart  (from  Bagadia  2006,  199) 


The  following  steps  are  the  most  commonly  addressed  regarding  CMMS 
implementation  (Wireman  2009,  107-121): 

•  Developing  a  project  plan  and  determining  resource  requirements  - 
Establish  implementation  teams  with  representatives  from  each  department 
affected  by  the  CMMS.  Establish  task-oriented  teams  responsible  for  the 
resources  and  tasks  associated  with  their  portion  of  the  CMMS 
implementation. 

•  Updating  all  current  records  -  Ensure  the  CMMS  data  is  factual  and  up-to- 
date  and  in  the  correct  format  for  input  to  the  system.  Typical  equipment  data 
includes:  general  information,  nameplate  data,  inspection  reading  limits, 
financial  information,  and  asset  history. 

•  Software  installation  -  Load  the  program  on  the  required  computers  and 
make  sure  it  works  correctly. 
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•  Data  entry  &  migration  -  Input  the  information  from  the  current  data  keeping 
system  into  the  CMMS  program  database.  Special  attention  needs  to  be 
placed  on  cataloging  and  data  accuracy. 

•  System  introduction  and  updates  -  Present  the  CMMS  program  to  the  users 
appropriately  in  a  manner  that  enables  them  to  use  it  as  required. 

•  CMMS  training  -  Often  overlooked,  training  is  indispensable  for  the  correct 
and  effective  use  of  the  CMMS. 

Any  CMMS  program  may  be  limited  to  act  as  a  maintenance  database  if  not 
integrated  within  the  organization  correctly.  Figure  7  is  DOD’s  representation  of  the  CBM+ 
concept  and  its  infrastructure  building  blocks.  A  well-implemented  CMMS  program  interacts 
with  all  CBM-i-  infrastructure  blocks. 


Figure  7.  CBM-i-  infrastructure  (from  Department  of  Defense  2008,  3-18) 

CMMS  introduction  and  training  are  two  key  steps  during  CMMS  introduction  as  a 
new  maintenance  tool  to  a  large  maintenance  organization.  A  CMMS  can  only  perform 
effectively  when  the  users  input  enough  timely  accurate  data  to  build  a  history  that  may 
indicate  equipment  trends.  Building  an  accurate  history  of  equipment  performance  may 
become  difficult  when  the  CMMS  users  are  resistant  or  poorly  trained  on  how  to  operate  the 
program. 
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2.  CMMS  Requirements  in  Official  CBM  Policy 

a.  Chief  of  Naval  Operations  ( CNO)  CBM  Policy 

The  CNO’s  CBM  policy,  released  in  December  2007,  establishes  the  policies  and 
responsibilities  related  to  the  implementation  of  CBM  on  naval  assets.  The  instruction 
mandates  a  fleet- wide  transition  to  incorporating  CBM  strategies  “to  the  maximum  extent 
possible”  into  all  maintenance  programs  while  continuing  to  employ  RCM  methods  for 
maintenance  planning  (Chief  of  Naval  Operations  2007,  1).  The  policy  requires  training  for 
maintenance  managers  and  technicians  as  well  as  the  integration  of  CBM  with  a  data 
collection  system. 

CBM  enabling  information  systems  (data  collection  and  information 
analysis)  shall  be  integrated  with  maintenance  management  and  logistics 
support  information  systems.  The  impact  of  information  systems  data 
collection,  processing,  and  warehousing  requirements  on  afloat  and  ashore 
resources  shall  be  considered  in  system  design,  development,  and  life-cycle 
planning.  (Chief  of  Naval  Operations  2007,  3) 

The  CNO  CBM  policy  explicitly  identifies  the  importance  of  a  suitable  information 
system  for  the  success  of  CBM  implementation. 

b.  NAVSEA  CBM+ Policy 

The  Naval  Sea  Systems  Command  (NAVSEA)  is  assigned  to  build,  deliver,  and 
maintain  Navy  ships.  NAVSEA  instruction  4790. 27A,  released  in  May  2013,  establishes  the 
policy  and  responsibilities  for  RCM,  CBM,  and  CBM+  integration  into  the  ship’s  lifecycle 
maintenance  strategy.  The  instruction  makes  CBM-i-  the  approved  maintenance  strategy  for 
ships,  ship  systems,  and  equipment  and  it  also  mandates  the  use  of  software  to  conduct 
CBM-i-  analysis  and  other  functions. 

Ensure  equipment  and  systems  performance  and  reliability  analysis  software 
and  display  systems  provide  timely  actionable  data  against  the  predictive 
failure  risk  models  during  acquisition  and  sustainment.  (Naval  Sea  Systems 
Command  2013,  10) 

NAVSEA  policy  explicitly  identifies  the  need  for  software  to  meet  the  fleet’s  CBM+ 

needs. 
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C.  CMMS  USE  IN  THE  LCS  PROGRAM 

The  LCS  program  office  tasked  two  civilian  contractors  with  executing  maintenance 
management  including  the  planning  and  execution  of  all  intermediate,  depot,  and  a  large 
percentage  of  the  organizational  level  maintenance.  Both  contractors  have  independently 
implemented  the  use  of  an  IBM  developed  CMMS  in  support  of  the  CBM+  strategy.  Navy 
maintenance  representatives  do  not  provide  specific  guidance  defining  how  the  contractor 
shall  use  CMMS  in  support  of  CBM+. 

The  CMMS  is  used  as  a  maintenance  database  and  PM  scheduling  tool.  CMMS  use 
varies  between  contractors.  Each  contractor  uses  the  CMMS  in  accordance  with  their 
independently  developed  maintenance  approach.  Some  differences  in  their  CMMSs  involve 
equipment  coverage,  program  applications,  data  generation,  and  general  program  usage. 
Maintenance  managers  and  supervisors  manually  input  the  equipment  status,  maintenance 
history,  and  maintenance  plans  into  the  CMMS.  Integration  of  the  current  CMMS  with  Navy 
legacy  maintenance  programs  has  been  challenging. 

D.  CBM+  MAINTENANCE  MANAGEMENT  SYSTEM  STAKEHOLDERS 

CBM+  maintenance  management  system  stakeholders  are  individuals  and 
organizations  interested  in  the  successful  employment  of  the  CMMS  as  a  CBM+  support 
system.  Navy  maintenance  stakeholders  include  government  and  civilian  organizations  and 
managers  involved  in  the  maintenance  process  throughout  the  life  of  a  ship. 

a.  Department  of  Defense 

DOD  establishes  CBM+  instructions  and  directives  for  military  departments  and 
defense  agencies. 

b.  Department  of  the  Navy 

The  CNO  establishes  policies  and  instructions  for  CBM+  implementation  and 
integration  onboard  naval  ships,  submarines,  aircraft,  equipment,  and  infrastructure. 
NAVSEA  is  the  Department  of  the  Navy  (DON)  command  responsible  for  promulgating, 
updating,  and  implementing  maintenance  policy  and  instruction. 
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c.  Program  Office 

Program  office  personnel  include  government  decision  makers  responsible  for  all 
aspects  of  life-cycle  management  to  include  maintenance  planning,  execution,  and 
budgeting. 

d.  CMMS  Power  Users 

Power  users  include  government  personnel  and  civilian  contractors  who  use 
advanced  CMMS  features  and  modules  to  perform  their  primary  tasks.  Power  users  are 
primarily  involved  in  the  business  and  execution  aspect  of  maintenance  and  are  not 
necessarily  programmers.  Their  tasks  include  supply  management,  data  analysis, 
maintenance  procedure  development,  and  other  related  to  information  technology  (IT). 

e.  Maintenance  Managers 

Maintenance  managers  include  government  and  contractor  personnel  responsible  for 
maintenance  planning,  execution,  reporting,  and  budgeting. 
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III.  LITERATURE  REVIEW:  MEASURES  OF  EFFECTIVENESS 


Measures  of  effectiveness  (MOE)  are  an  integral  systems  engineering  (SE)  and 
general  management  tool,  yet  a  universal  MOE  definition,  across  or  within  management 
disciplines,  does  not  exist.  MOEs  are  usually  discussed  in  SE,  business,  and  management 
literature.  Most  of  the  engineering  and  non-engineering  literature  addressing  MOEs  lacks 
depth  in  their  explanation  of  MOE  development.  MOEs  are  generally  described  as  sets  of 
required  measurements  that  aid  in  determining  how  well  something  achieves  its  operational 
purpose.  Most  sources  describe  the  importance  of  MOEs,  the  features  that  make  good  MOEs, 
and  they  often  provide  insight  into  their  applications.  However,  there  is  a  lack  of  literature 
seeking  to  present  a  repeatable  MOE  development  method  or  framework. 

Roedler  and  Jones  (2005,  9)  define  MOEs  as  “operational  measures  of  success  that 
are  closely  related  to  the  achievement  of  mission  or  operational  objectives”.  In  this  analysis 
MOEs  are  a  subset  of  technical  measures  used  to  develop  technical  performance  measures 
providing  a  system’s  technical  resolution  and  insight.  They  are  identified  by  the  stakeholder 
early  in  the  acquisition  process  with  the  goal  of  providing  insight  into  specific  operational 
requirements.  An  illustration  of  these  interdependent  measures  can  be  seen  in  Eigure  8. 


Technical 

Insight 

(Progress 

ARisk) 


Eigure  8.  Technical  measures  relationships  (from  Roedler  and  Jones  2005,  15) 
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Roedler  and  Jones  provide  a  set  of  guidelines  that  describe  the  characteristics  that 
make  a  well  selected  MOE  advantageous  to  the  user  and  the  analyst.  A  summary  of  these 
guidelines  is  included  below  in  Table  2. 


Table  2.  MOE  selection  guidance  (from  Roedler  and  Jones  2005,  36) 


MOE 

Provides  insight  into  at  least  one 
operational  objective  or  mission 
requirement _ 

MOEs  should  not  be  strongly 
correlated:  provide  insight  into 
different  aspects  of  the  operational 
alternative _ 

Select  and  define  in  the  context  of 
the  operational  objective  :  no 
predefined  MOEs/values _ 

Select  and  define  independent  of 
the  alternatives  at  hand  :  represent 
an  independent  means  to 
collectively  evaluate  the 
alternatives _ 

Select  only  a  few  MOE/MOPs  : 
may  be  an  order  of  magnitude 
more  TPMs _ 

Each  KPP  should  have  an 
associated  MOE  or  MOP 


Roedler’s  analysis  provides  a  useful  qualitative  approach  to  MOE  development  but  it 
lacks  a  methodical  framework.  His  guidelines  are  useful  to  revisit  during  the  MOE 
development  process  because  they  are  good  descriptors  of  verifiable  measurements. 

Wireman  addresses  five  related  performance  indicators  for  a  corporation.  Eike 
Roedler,  Wireman  considers  MOEs  to  be  part  of  a  larger  framework  of  measures.  The 
complete  framework  becomes  the  performance  indicator  pyramid.  Each  level  in  the  pyramid 
contains  performance  indicators  at  five  different  levels  of  the  corporation.  The  principal 
performance  indicators  are  defined  by  corporate  management  at  the  pyramid’s  apex.  All 
other  performance  indicators  in  the  corporation  are  shaped  by  those  at  the  top  of  the  pyramid. 
Wireman  argues  that  when  the  indicators  are  developed  from  the  bottom  up  they  may 
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become  conflictive  and  unsupportive  of  the  corporate  vision.  The  performance  indicator 
pyramid  is  displayed  in  Figure  9. 


Coipoiate 

Indicators 


Financial 

Perfonnance 

Indicators 


Efficiency  and 
Eli'ectheness 
Performance  Indicators 


Tactical  Performance  Indicators 


Functional  Performance  Indicators 


Figure  9.  Wireman's  performance  indicator  pyramid  (after  Wireman, 
Developing  Performance  Indicators  for  Managing  Maintenance 

2005,  220) 


Although  a  relationship  exists  between  the  five  levels  in  the  pyramid,  the  middle  and 
bottom  levels  address  the  traditional  MOEs.  The  middle  level  links  the  measurements  of 
efficiency  and  effectiveness.  Efficiency  describes  how  well  a  task  is  performed  while 
effectiveness  describes  how  well  the  task  meets  its  goals  and  requirements.  Wireman 
emphasizes  using  a  system’s  output  measures  instead  of  the  input  measures  to  develop  the 
results  required  to  determine  the  efficiency  and  effectiveness  of  a  task.  Output  measures  are 
associated  with  efficiency  and  effectiveness  indicators.  Input  measures  and  process  measures 
seek  to  describe  workload.  In  this  lean-derived  approach  the  effectiveness  indicators  are 
chosen  to  support  financial  indicators  seeking  to  reduce  waste,  expense,  and  effort 
(Wireman,  Developing  Performance  Indicators  for  Managing  Maintenance  2005,  210). 

Wireman’s  indicators  are  expressed  as  percentages  or  ratios.  Some  of  the  work-order 
system  effectiveness  indicators  include  percentages  of  work  distributon  by  type  of  work- 
order.  Wireman  expresses  some  of  these  percentages  as  (Wireman,  Developing  Performance 

Indicators  for  Managing  Maintenance  2005,  211): 
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Emergency  Work  Orders 
Total  Work  Orders 

Preventive  Work  Orders 
Total  Work  Orders 

Corrective  Work  Orders 
Total  Work  Orders 

The  functional  performance  indicators  provide  ratios  used  to  evaluate  the 
performance  of  specific  maintenance  functions.  Some  of  the  CMMS  functional  performance 
indicators  include  (Wireman,  Developing  Performance  Indicators  for  Managing  Maintenance 
2005,  217-218): 

Total  Number  of  Equipment  Items  in  CMMS 
Total  Number  of  Equipment  Items  in  the  plant 

Total  Number  of  Part  Items  in  CMMS 
Total  Number  of  Part  Items  in  the  plant 

Total  Number  of  PM  tasks 
Total  Number  of  Equipment  Items  in  the  plant  x  3 

Wireman  provides  close  to  100  performance  ratios  that  a  maintenance  manager  may 
use  to  develop  an  initial  set  of  performance  indicators.  Each  ratio  measures  performance  at  a 
different  level  in  the  corporation.  However,  Wireman  fails  to  provide  explanations  as  to  the 
significance  or  development  of  these  ratios  as  they  determine  levels  of  performance.  Using 
performance  indicators  without  justification  makes  an  organization  data  rich  but  information 
poor.  Wireman  also  fails  to  address  stakeholder  needs  or  requirements  elicitation.  One  could 
assume  that  the  corporate  indicators  at  the  pyramid’s  apex  may  define  needs  and 
requirements  but  this  approach  may  lead  to  MOE  development  based  on  wants  instead  of 
needs.  Using  corporate  indicators  in  MOE  development  may  result  in  inappropriate 
measurements  that  are  solution  specific,  alternative  specific,  and  dependent  on  corporate 
opinion. 
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Stevens  describes  MOEs  in  the  context  of  operational  testing  and  evaluation  (OT&E). 
In  OT&E,  MOEs  are  defined  as  “any  set  of  criteria  established  to  determine  the  resolution  of 
a  critical  issue”  (Stevens  1986,  54).  Stevens’  MOE  development  rules  are  specific  for  a 
testing  environment  but  remain  useful  and  transferable  for  other  uses.  OT&E  MOEs  are  tied 
to  critical  issues  which  may  be  broken  down  into  lower  level  elements  to  provide  combinable 
specific  data.  Stevens’  testing  viewpoint  permits  the  use  of  MOEs  to  understand  a  system’s 
test  results  and  develop  a  set  of  conclusion  data  that  may  be  used  for  other  system’s  tests. 

Stevens  lists  the  following  characteristics  to  describe  a  good  MOE  (Stevens  1986, 
55): 

•  The  MOE  should  be  relevant. 

•  The  set  of  MOEs  should  be  complete. 

•  The  MOE  should  be  precisely  defined. 

•  The  MOE  sets  should  be  mutually  exclusive. 

•  The  MOE  should  be  expressed  in  terms  that  are  meaningful  to  testers  and 
developers. 

•  MOE  meaning  should  not  be  open  to  interpretation  with  the  passage  of  time. 

•  MOE  inputs  should  be  measureable. 

Stevens  also  provides  seven  MOE  development  ground  rules  (Stevens  1986, 55-56): 

•  There  should  be  one  MOE  for  each  mission  capability. 

•  MOE  weights  shall  be  assigned  by  decision  makers. 

•  Eully  define  the  mission  and  scenario  before  collecting  measurements  during 
testing. 

•  Measurements  should  not  interfere  with  system  operation. 

•  State  MOE  quantitative  measurements  as  probabilities. 

•  All  qualitative  measurements  should  use  the  same  standard. 

•  When  recording  system  failures  during  testing  include  both:  faulty  system 
and  hardware  failures. 

Many  of  Stevens’  MOE  characteristics  and  development  ground  rules  echo  those 
discussed  by  Roedler.  But  like  Roedler,  Stevens  succeeds  in  describing  good  MOEs  and  their 
relevance  to  an  application  but  fails  to  present  a  development  method  or  framework. 
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Sproles  acknowledges  the  lack  of  both  a  universal  MOE  definition  and  a  method  to 
formulate  them.  MOEs  and  measures  of  performance  (MOP)  are  commonly  used  as 
interchangeable  terms  in  management  and  engineering  literature.  Sproles  uses  developmental 
viewpoints  to  differentiate  between  MOEs  and  MOPs. 

The  distinction  between  effectiveness  and  performance  shows  that  MOEs  and 
MOPs  are  formulated  from  different  viewpoints.  An  MOE  refers  to  the 
effectiveness  of  a  solution  and  is  independent  of  any  particular  solution;  an 
MOP  refers  to  the  actual  performance  of  an  entity.  The  relationship  between 
the  two  types  of  measures  can  be  described  using  the  analogy  of  the 
relationship  between  effectiveness  and  efficiency.  Effectiveness  is  how  well 
something  does  its  job.  Efficiency  is  how  well  something  does  what  it  is 
doing.  Therefore,  efficiency  can  be  high  while  effectiveness  is  low  or  even 
zero.  Something  can  be  done  well  even  though  it  is  the  wrong  job  which  is 
being  done.  An  MOE  will  indicate  a  property  which  a  potential  solution  must 
possess  in  order  to  meet  a  need:  An  MOP  will  tell  what  something  is  capable 
of  doing,  even  if  this  is  not  necessarily  what  the  stakeholders  want  it  to  do. 

The  difference  between  effectiveness  and  performance  as  applied  to  a 
solution  to  a  need  is  that  effectiveness  is  a  quality  of  fitness  for  service  or  of 
producing  the  results  for  which  it  was  intended.  Performance  is  the  quality  of 
.doing  something,  and  doing  something  does  not  necessarily  indicate  fitness 
for  service.  (Sproles,  Coming  to  Grips  with  Measures  of  Effectiveness  2000, 

56-57) 

Sproles’  places  stakeholder  needs  and  requirements  at  the  center  of  developing  a 
process  that  captures  how  well  a  system  performs  its  mission.  The  process  begins  with 
stakeholder  requirements  elicitation.  These  requirements  will  provide  the  characteristics  to 
determine  the  effectiveness  of  a  system  and  will  consequentially  influence  the  system 
selection  decisions.  This  method  combines  the  stakeholder’s  input  and  the  stakeholder 
viewpoint  to  identify  a  mission  to  frame  the  MOEs.  Eike  Stevens,  Sproles  explains  that 
MOEs  need  to  address  critical  operational  issues  (COI)  exclusively.  A  COI  is  “an  emergent 
property  that  the  system  must  have  in  order  to  perform  its  function  [and]  that  a  solution  to  a 
need  must  possess  in  order  to  meet  the  need”  (Sproles,  Eormulating  Measures  of 
Effectiveness  2002,  256-257).  Developers  need  to  identify  the  system’s  critical  items  of 
interest  before  drafting  the  MOEs  to  help  determine  the  system’s  essential  operational 
attributes.  MOE  drafting  requires  creative  and  knowledgeable  people  with  good  management 
techniques  to  provide  answers  and  heuristics  that  can  be  honed  into  testable  and  measurable 
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statements  that  address  the  COIs.  Once  established,  the  MOEs  shall  be  evaluated,  iterated, 
and  revised  (Sproles,  Formulating  Measures  of  Effectiveness  2002,  258-259). 

Sproles’  MOE  development  process  is  illustrated  in  Figure  10. 


Figure  10.  MOE  development  process  (from  Sproles,  Formulating  Measures  of 

Effectiveness  2002,  255) 

Once  the  COIs  are  decided  the  MOE  follows  an  iteration  loop  to  ensure  it  remains 
relevant  to  the  system’s  measurements.  However,  if  the  COI  is  decided  by  a  viewpoint  or 
mission  that  is  no  longer  relevant  or  was  originally  mistakenly  determined  the  irrelevant 
MOE  will  be  stuck  in  the  iteration  loop  and  will  create  waste. 

Sproles’  MOE  formulating  process  serves  as  a  good  starting  point  for  a  way  towards 
a  methodical  MOE  development  approach;  however  it  is  too  subjective  to  the  input  of  a 
singular  stakeholder  because  it  narrows  the  solution  trade  space  by  possibly  limiting  MOE 
selection  decision  to  one  viewpoint.  A  narrow  solution  trade  space  may  lead  to  missing 
alternatives  that  may  meet  the  needs.  Sproles  links  MOEs  to  COIs  and  COIs  to  needs  but 
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provides  no  guidance  in  differentiating  between  a  need  and  a  want.  Although  it  has  some 
faults,  Sproles’  stakeholder  centric  MOE  development  method  is  a  step  in  the  right  direction 
of  establishing  a  repeatable  method  to  develop  MOEs. 

Eeibowitz  discusses  MOEs  from  a  metaphysical  operation  analyst’s  viewpoint.  His 
approach  resonates  with  Sproles’  call  for  creativity  and  heuristics  in  the  formulation  of 
MOEs.  “A  measure  of  effectiveness  resembles  a  moral  principle  in  that  its  validity  cannot  be 
established  by  reason  alone.  We  must  make  a  value  judgment.  We  must  play  it  by  feel” 
(Eeibowitz  1958,  127).  Eeibowitz’  calls  his  qualitative  approach  to  choosing  MOEs  the 
method  of  dynamic  compromise.  The  method  of  dynamic  compromise  is  comprised  of  four 
steps: 

•  Obtain  an  approximate  measure  of  the  system’s  effectiveness  using  the 
limited  understanding  of  the  supersystem  (Eeibowitz  1958,  130). 

•  Adjust  the  measure  and  relate  it  to  the  system’s  elements  (Eeibowitz  1958, 
130). 

•  Readjust  the  measure  until  it  is  satisfactory  to  the  decision  maker  (Eeibowitz 
1958,  130). 

•  Readjust  the  measure  until  the  projected  study  does  not  exceed  time  and 
effort  deadlines  (Eeibowitz  1958,  130). 

Once  the  analyst  readjusts  the  fourth  step  measure  they  compare  it  to  the  true  needs 
of  the  system  in  question.  This  last  step  is  where  the  analyst  uses  his  “feel”.  If  the  fourth 
measure  is  not  close  to  a  true  need  the  analyst  has  five  courses  of  action: 

•  Learn  more  about  the  supersystem  (Eeibowitz  1958,  130). 

•  Learn  more  about  the  system  itself  (Eeibowitz  1958,  130). 

•  Talk  the  decision  maker  into  revising  his  interpretation  (Eeibowitz  1958, 
130). 

•  Extend  the  scope  of  the  study  (Eeibowitz  1958,  130). 

•  Cancel  the  study  (Eeibowitz  1958,  130). 

Eeibowitz’  method  of  dynamic  compromise  is  simple  and  maintains  the  development 
process  grounded  on  the  fact  that  an  MOE  “must  be  reasonably  close  to  representing  the  true 
purpose  of  the  system.  If  it  is  not  then  all  the  linear  programming  and  all  the  game  theory  in 
the  world  will  not  save  us  from  optimizing  auto  assembly  lines  so  as  to  provide  the 
maximum  number  of  coffee  breaks  per  hour”  (Eeibowitz  1958,  130). 
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Essawy  and  Zein-sabatto  define  MOEs  and  MOPs  specifically  for  CBM  systems. 
There  is  a  need  for  CBM  technology  developers  to  develop  a  set  of  universal  measures  for 
competing  systems  comparison,  suitability  determination  and,  fitness-for-purpose  evaluation. 
Essawy  and  Zein-sabatto  use  the  MOE  and  MOP  terms  almost  interchangeably,  only 
differentiating  between  them  by  the  scope  of  the  measured  attribute.  MOEs  address  those 
measurements  specific  to  a  system  while  MOPs  address  the  measurements  specific  to 
subsystems  (Essawy  and  Zein-Sabatto  2000,  303). 

Essawy  presents  and  analyzes  several  mathematical  expressions  representing  CBM 
system  variables  considered.  These  variables  serve  as  inputs  for  functions  that  describe  the 
MOPs  or  MOEs.  Some  of  Essawy’ s  essential  variables  have  been  selected  and  are  explained 
below. 

Success  and  failure  rates  measure  the  rate  of  a  system’s  successful  function 
performance.  Is,  If,  and  It  are  the  number  of  success,  failure,  and  total  instances  respectively 
(Essawy  and  Zein-Sabatto  2000,  304). 

Success  Rate  =  — 

It 

Eailure  Rate  =  — 

It 

The  time  delay  is  the  time  required  by  a  system  to  perform  its  intended  function.  Tf 
and  Ti  are  the  final  and  initial  recorded  times  respectively  (Essawy  and  Zein-Sabatto  2000, 
304). 

Time  Delay  = 

Reliability  measures  the  frequency  of  system  failures.  The  average  extent  is  the 
average  partial  system  failure.  Average  extent  is  “a  summation  of  the  extent  of  all  of  the 
individual  partial  failures  divided  over  the  number  of  partial  failures  multiplied  by  the 
maximum  predicted  partial  failure  extent”  (Essawy  and  Zein-Sabatto  2000,  304).  The 
function  variables  are  the  total  number  of  partial  failures  (n)  and  the  extent  of  partial  failure 
at  each  incident  (A,)  (Essawy  and  Zein-Sabatto  2000,  304). 
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Average  Extent  =  —  V 

n  Max{A. ) 

Reliability  =  e^^'^  ^  “ 

Robustness  measures  the  ability  of  a  system  to  function  satisfactorily  in  the  presence 
of  changing  conditions  and  inputs.  I#  are  the  number  of  instances  in  which  a  changing 
condition  was  introduced.  The  index  variables,  i,  j,  are  the  various  conditions  that  can 
introduce  change  and  inputs  into  the  system  (Essawy  and  Zein-Sabatto  2000,  305). 

^  Ii  +I2  +  I3—  +  In 

Robustness  = -  V  (Success  Rate),.  , 

T+T+T  +T  ^ 

-*-1  '  ^2  ~  -‘-3***  ~  -‘-jsj  /,y,m,...z=l 

Essawy’ s  method  uses  the  results  of  these  variable  functions  to  calculate  a 
quantitative  MOE  or  MOP.  The  resulting  calculations  are  further  manipulated  using  one  of 
three  approaches: 

•  Weighed  sum:  Calculates  an  MOE/MOP  figure  by  using  the  sum  of  the 
products  of  the  calculated  variables  by  an  assigned  weight  corresponding  to 
the  variable’s  relevance  in  determining  successful  or  effective  performance 
(Essawy  and  Zein-Sabatto  2000,  307). 

•  Euzzy  logic :  This  method  uses  the  input  variables  to  emulate  human-decision 
making  methods  (Essawy  and  Zein-Sabatto  2000,  308). 

•  Neural  network  system:  The  MOE/MOP  figure  is  estimated  by  measuring  the 
“output  of  a  feed-forward  backpropagation  neural  network”  using  the 
calculated  variables  as  inputs.  Essawy  does  not  provide  an  example  of  this 
method  (Essawy  and  Zein-Sabatto  2000,  308). 

Essawy  and  Zein-sabatto  present  replicable  methods  to  calculate  MOEs  and  MOPs. 
Like  Wireman’s,  this  method  succeeds  in  providing  guidance  of  possible  effectiveness  and 
performance  indicators  but  fails  in  providing  a  reason  as  to  why  these  indicators  are  relevant. 
Essawy’ s  research  also  leaves  out  the  stakeholder  needs  and  requirements.  Also,  because 
Essawy  interchanges  the  terms  MOE  and  MOP,  some  of  his  variables  become  so  generic  that 
they  fail  to  capture  a  system’s  success  in  meeting  a  set  of  needs. 

Langford  defines  MOEs  as  “the  single-most  often  touted  and  applied  method  of 
thought  of  how  well  one  is  doing”  (G.  Langford,  Building  the  Determinants  of  Technology 
Effectiveness  2014,3).  Effectiveness  determination  depends  on  the  identification  of 
significant  measures  of  causality,  the  interpretation  of  measurements,  and  the  measurement 
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viewpoints;  these  three  factors  allow  for  the  determination  of  functions,  processes,  effects, 
and  data  evaluation  as  they  relate  to  measurable  effectiveness.  Langford  proposes  developing 
MOEs  using  an  integrative  framework  (G.  Langford,  Building  the  Determinants  of 
Technology  Effectiveness  2014,  4-7).  The  framework  is  a  3x3  grid  composed  of  an 
objective  frame  and  a  subjective  frame.  The  objective  frame  captures  the  product  or  service; 
the  subjective  frame  captures  the  management  process  to  satisfy  the  product  or  service.  The 
intersections  of  frame  elements  create  nine  cardinal  points  that  represent  the  MOE  domains. 
These  cardinal  points  are  the  interaction  between  a  product’s  constituents  that  determine 
overall  system  effectiveness.  An  MOE  set  that  captures  all  the  necessary  objectives  required 
for  success  shall  cover  all  domains  in  the  framework.  Figure  1 1  displays  the  framework’s 
two  frames,  the  domain  descriptions,  and  domain  elements. 
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Integrative  framework  (from  G.  Langford,  Determinants  of 
Deterrence  Effectiveness  2014,  7) 


The  use  of  this  framework  as  an  MOE  development  tool  emphasizes  the  process - 
object  relationships  used  to  evaluated  the  system’s  fitness-for-purpose.  As  illustrated  in 
figure  12,  fitness-for-purpose  is  the  real  knowledge  required  to  determine  how  well  a  system 
satisfies  its  objective  (G.  Langford,  Building  the  Determinants  of  Technology  Effectiveness 
2014,8). 
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Linked  to  Liiilced  to 

Figure  12.  Linkage  between  MOEs  and  fitness-for-purpose  (from  G.  Langford, 
Building  the  Determinants  of  Technology  Effectiveness  2014,  8) 

Eangford’s  MOE  development  framework  does  not  dictate  a  set  of  suggested 
measurements  adaptable  to  any  system.  The  MOE  set  produced  by  this  framework  is  unique 
in  that  it  addresses  the  specific  requirements  to  satisfy  the  system  managerial  and  operational 
specific  needs.  Langford’s  method  also  calls  for  periodic  feedback  and  iteration  to  ensure 
effectiveness  measurements  remain  accurate  as  the  system’s  objects  interactions  change  with 
time  and  experience. 
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IV.  METHODOLOGY 


The  computerized  maintenance  management  system  (CMMS)  measures  of 
effectiveness  (MOE)  were  developed  using  traditional  system  engineering  (SE)  concepts  and 
a  nine-step  method  for  characterization.  The  SE  concepts  are  standard  in  the  identification  of 
user  needs,  analyzing  system  requirements  and  functional  requirements. 

A.  NINE-STEP  METHOD 

MOEs  were  characterized  using  a  modified  application  of  Langford’s  nine-step 
method  that  allows  the  MOE  development  concept  to  become  repeatable,  and  able  to  be 
validated  with  operational  and  managerial  needs  (G.  Langford,  Determinants  of  Deterrence 
Effectiveness  2014, 7- 12).  A  flow  diagram  of  the  nine  step  method  is  displayed  in  Eigure  13: 


Eigure  13.  Nine  step  methodology  to  characterize  MOEs  (from  G.  Langford, 
Determinants  of  Deterrence  Effectiveness  2014,  12) 
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This  research  only  used  the  first  seven  steps  of  the  nine-step  method.  The  last  two 
steps  are  better  used  during  MOE  reevaluation.  The  seven  steps  used  in  this  study  and  their 
application  to  CMMS  MOE  development  are  described  below  (G.  Eangford,  Determinants  of 
Deterrence  Effectiveness  2014,  7-12): 

•  Define  terminology:  Defined  CMMS,  CBM-i-,  and  other  maintenance 
management  working  terms  to  determine  the  research  scope. 

•  Delineate  boundaries  and  functions:  Performed  a  functional  analysis  of  the 
key  measurable  CMMS  and  management  functions. 

•  Perform  lifecycle  analysis:  Defined  the  life  cycle  of  the  key  measurable 
CMMS  and  management  functions. 

•  Define  requirements :  Derived  stakeholder  requirement  by  analyzing  solicited 
stakeholder  needs. 

•  Postulate  solution  set:  Developed  a  set  of  solutions  to  satisfy  the  problem 
domain  issues. 

•  Determine  theoretical  foundations:  Applied  management  maintenance  and 
logistics  theory  to  the  problem  issue. 

•  Eormalize  framework:  Mapped  subjective  elements  to  objective  elements 

using  the  integrative  framework. 

1.  Define  Terminology 

A  set  of  selected  CMMS  MOE  terms  were  defined.  The  terms  were  chosen  based  on 
their  quality  to  bound  and  scope  the  work  within  the  problem  (G.  Eangford,  Determinants  of 
Deterrence  Effectiveness  2014). 

2.  Delineate  Boundaries  and  Functions 

The  key  functions  of  the  function  ‘To  Manage’  were  analyzed  and  delineated.  The 
functional  decomposition  facilitated  the  MOE  development  by  providing  specificity  to  the 
multiple  sub-functions  involved  in  management. 

3.  Perform  Lifecycle  Analysis 

This  section  analyzes  the  lifecycle  of  a  work-order  as  it  relates  to  maintenance 
management.  The  maintenance  management  lifecycle  was  derived  from  the  traditional  Navy 
surface  maintenance  process  described  in  the  Joint  Elect  Maintenance  Manual.  This  lifecycle 
analysis  clarifies  the  scope  of  the  problem. 


38 


4. 


Define  Requirements 


CMMS  requirements  were  derived  from  stakeholders’  CMMS  needs.  The 
stakeholders’  needs  were  derived  from  condition-based  maintenance  plus  (CBM-i-)  policy 
documents,  LCS  concept  of  operations,  and  supplemented  by  input  from  CMMS  power 
users.  Department  of  Defense  (DOD)  and  Navy  CBM-i-  policy  documents  and  instructions 
provide  a  set  of  specified  needs  and  general  descriptions  of  the  expected  accomplishments 
and  gains  resulting  from  the  correct  imlpementation  of  the  maintenance  approach.  CMMS 
power  users  included  maintenance  managers  belonging  to  civilian  contractors  and 
government  personnel.  The  power  users  provided  real  needs,  perceived  needs,  and  expressed 
needs  as  they  relate  to  their  experience  at  specific  jobs  and  tasks  including  supply 
management,  maintenance  planning,  information  technology  (IT)  analysis,  and  reliability 
engineering.  Some  of  the  needs  provided  by  users  were  motivated  by  the  set  of  realized 
needs  resulting  from  the  current  CMMS  in  use.  The  overarching  set  of  needs  was  analyzed, 
prioritized,  and  developed  into  CMMS  operational  and  functional  requirements.  These 
requirements  were  later  used  for  MOE  traceability  and  validation. 

5.  Postulate  Solution  Set 

A  solution  set  based  on  the  CMMS  requirements  and  problem  scoping  was  proposed. 

6.  Determine  Theoretical  Foundations 

Applicable  theories  of  social  behavior  were  applied  to  the  situation  for  CMMS  usage. 
The  theories  selected  as  applicable  to  this  problem  were  management  theory  and  decision 
theory. 


7.  Formalize  Integrative  Framework 

The  MOE  development  framework  was  derived  from  Langford’s  integrative 
framework  (G.  O.  Langford  2012,  81-99).  The  framework  captures  the  product  and 
management  needs  in  the  nexus  formed  by  the  intersections  of  an  objective  frame  and  a 
subjective  frame.  The  intersections  of  frame  elements  create  nine  cardinal  points  that 
represent  the  MOE  domains.  An  MOE  set  that  captures  all  the  necessary  objectives  required 
for  success  shall  cover  all  domains  in  the  framework.  Eigure  14  displays  the  framework’s 
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two  frames,  the  domain  descriptions,  and  domain  elements.  The  arrows  in  Figure  14  indicate 
the  interaction  sequence  between  the  product  or  service  in  the  objective  frame  and  the 
process  in  the  subjective  frame.  The  sequence  begins  with  cognitive  structures,  completing  a 
cardinal  point  before  moving  to  the  next.  The  following  is  a  typical  sequence  to  navigate  the 
framework  (G.  O.  Langford  2012,  88): 

•  Cognition-Objects,  Cognition-Functions,  Cognition-Behaviors 

•  Procedures-Objects,  Procedures-Functions,  Procedures-Behaviors 

•  Models-Objects,  Models-Functions,  Models-Behaviors 


Figure  14.  Integrative  framework  (from  G.  Langford,  Determinants  of 

Deterrence  Effectiveness  2014,  7) 


Figure  15  displays  the  framework  with  descriptions  of  the  domain  elements  and  the 
cardinal  points  created  at  each  intersection. 
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Figure  15.  Integrative  framework  with  MOE  domain  description 

(from  G.  O.  Langford  2012,  89) 


Figure  16  displays  the  framework  with  descriptions  of  the  specific  MOEs  that  result 
from  the  process  object  relationships  at  each  cardinal  point. 
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exchange  of  EMMI  (MOE-g):  & 
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properties,  traits,  and  attributes  (MOE- 
0);  Models  or  representations  of  all 
social,  pohtical.  economic  elements 
(MOE-j) 

Figure  16.  MOE  Framework  Cardinal  Points  (from  G.  Langford  Building  the 
Determinants  of  Technology  Effectiveness  2014,  12) 
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The  user  needs,  theoretical  foundations,  and  system  and  functional  requirements  were 
used  to  map  the  cognitive  and  procedure  processes  to  physical  objects  in  the  integrative 
framework  to  develop  the  MOEs. 

This  analysis  focused  on  the  lower  left  2x2  grid  including  Cognition-Object, 
Cognition-Function,  Procedure- Object,  and  Procedure-Function.  The  ‘Models  & 
Represenatations’  domain  requires  to  be  completed  using  a  specific  system  and  organization 
to  provide  the  necessary  architecture  elements  for  modeling.  The  purpose  of  this  study  is  to 
generate  a  set  of  MOEs  that  is  applicable  to  any  maintenance  management  software.  The 
domain  is  outside  of  the  scope  of  this  study.  A  framewok  tailored  to  this  study’s  scope  is 
displayed  in  FigurelV. 
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Functions 

Prognostication  of  consequences  of 
interactions  between  objects  tluough 
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Experience  with  posited  objects  (MOE- 
e)  &  anticipated  responses  of  posited 
objects  (MOE-r) 

Availability  and  Validity  of  processes 
and  inechanisms  resulting  in  the 
selection  and  development  of  all 
physical  elements  (MOE-s): 

Processes  and  mechanisms  resulting  in 
the  development  of  all  physical 
elements  and  operational  contexts 
(MOE-x) 

Figure  17.  Modified  Integrative  Framework  (from  G.  Fangford,  Building  the 
Determinants  of  Technology  Effectiveness  2014,  12) 


Once  developed,  the  MOEs  were  validated  by  tracing  them  to  the  CMMS  user 
requirements. 
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V.  MEASURES  OF  EFFECTIVENESS  DEVELOPMENT 


This  chapter  follows  the  development  of  a  set  of  eight  MOEs  using  the  nine  step 
methodology.  The  MOEs  capture  the  fitness-for-purpose  of  a  CMMS  implementation. 

A.  TERMINOLOGY  DEFINITION 

CMMS  and  maintenance  terminology  was  defined  in  Chapters  I,  II,  and  III.  A  list  of 
selected  defined  terms  that  bound  the  scope  of  the  problem  is  included  in  this  section. 

•  Maintenance:  The  administrative  and  technical  actions  required  to  maintain 
an  item  in  a  condition  in  which  it  can  perform  its  intended  function  (INCOSE 
1998,  137).  Maintenance  needs  define  the  maintenance  management 
approach. 

•  Reliability  centered  maintenance  (RCM):  A  logical  function-oriented  process 
that  identifies  optimal  equipment  maintenance  requirements  based  on 
reliability  characteristics  and  operating  conditions.  Its  goal  is  to  eliminate  PM 
tasks  that  do  not  add  value  to  the  maintenance  process  by  using  reliability 
data  to  adjust  maintenance  time  intervals. 

•  Preventive  maintenance  (PM) :  A  time-driven  maintenance  strategy  based  on 
calendar  time,  equipment  operating  time,  or  a  number  of  cycles  intended  to 
prevent  unscheduled  downtime  or  damage  that  may  lead  to  a  catastrophic 
failure  (Department  of  Defense  2008,  2-2). 

•  Condition-based  Maintenance  Plus  (CBM-i-) :  An  RCM  derived  maintenance 
strategy  that  uses  real  time  or  near  real  time  system  reliability  sensor  data, 
and  other  RCM  methods  to  determine  the  best  maintenance  requirements  for 
a  system. 

•  Corrective  maintenance  (CM):  Reactive  maintenance  that  includes  all 
unscheduled  maintenance  actions  executed  as  a  result  of  system  malfunction, 
failure,  or  deterioration  (Navy  Personnel  Command  2013). 

•  Computerized  maintenance  management  system  (CMMS):  Eundamental 
CBM  database  tools  that  assists  maintenance  activities  in  decision  making 
regarding  planning,  management,  and  administration  of  maintenance  actions. 
The  CMMS  is  an  interactive  equipment  parameter  and  condition  database 
that  can  receive  data  input  from  users  or  from  sensors  and  meters.  The 
equipment  data  is  stored  as  a  system-specific  maintenance  history  that  may 
be  used  by  decision  makers  to  detect  trends  and  perform  analysis. 

•  Measures  of  effectiveness  (MOE):  A  combination  of  measures  intended  to 
determine  to  what  extent  objectives  are  accomplished  and  how  well  the 


43 


results  compare  with  the  desired  results  (G.  Langford,  Building  the 
Determinants  of  Technology  Effectiveness  2014,  5). 

B.  MAINTENANCE  MANAGEMENT  FUNCTIONS  DELINEATION 

The  functions  of  maintenance  management  are: 

•  To  plan 

•  To  communicate 

•  To  organize 

•  To  direct 

•  To  control 

Maintenance  managers  employ  these  functions  via  the  CMMS  program  to  create 
synergy  amongst  the  maintenance  team  members  including  contract  specialists,  planners, 
maintainers,  and  other  managers.  Figure  18  displays  the  functional  hierarchy  of  the  function: 
“To  Manage  Maintenance”. 
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Figure  18.  Maintenance  management  functional  hierarchy 
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1. 


To  Plan 


The  function  ‘to  plan’  includes  the  available  CMMS  activities  required  to  conduct 
necessary  maintenance  planning.  Traditionally,  the  planning  function  is  fundamental  to  any 
CMMS  program  because  it  predicts  and  helps  organize  the  required  data  used  in  its  modules. 

a.  Plan  Maintenance  Execution 

Maintenance  execution  planning  includes  planning  the  required  equipment 
maintenance  actions  and  repairs.  Maintenance  execution  planning  considers  the  availability 
of  the  correct  tooling,  parts,  personnel,  maintenance  authorizations,  and  coordination  within 
maintenance  activities. 

The  CMMS  should  provide  execution  planning  information  in  the  work-order  data 
and  the  logistics  and  scheduling  modules. 

b.  Plan  Maintenance  Budgeting 

Maintenance  budgeting  planning  includes  calculating  and  estimating  the  required 
funds  for  upcoming  maintenance.  The  CMMS  provides  a  list  of  upcoming  required 
maintenance  data  that  includes  required  parts,  estimated  duration  of  repair,  personnel 
requirements,  and  equipment  nomenclature. 

c.  Plan  Maintenance  Logistics 

Maintenance  logistics  planning  includes  estimating  the  required  parts  and 
consumables  supply  and  all  shipping  and  transportation  planning  for  maintenance. 

d.  Plan  Maintenance  Scheduling 

Maintenance  scheduling  planning  includes  the  planning  and  coordination  of  the 
maintenance  availability  scheduling  considering  supply  and  parts  delivery  times,  foreseen 
delays,  and  integrated  work  planning. 

e.  Plan  Maintenance  Required  Personnel 

Maintenance  personnel  requirement  planning  includes  the  planning  and  coordination 
of  available  maintenance  personnel  to  complete  the  required  maintenance  within  schedule. 
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2. 


To  Communicate 


The  function  ‘to  communicate’  includes  the  available  CMMS  activities  required  for 
transferring  maintenance  management  information  within  and  outside  of  the  organization. 
All  entities  involved  in  the  maintenance  management  efforts  share  a  common  picture  when 
the  CMMS’  communicating  function  is  employed  effectively.  Managerial  situational 
commonality  is  fundamental  for  swift  and  effective  decision  making. 

a.  Communicate  Maintenance  Planning 

Maintenance  planning  communications  include  the  transfer  of  all  information 
required  to  create  a  common  situational  picture  to  facilitate  planning. 

b.  Communicate  Maintenance  Budget 

Maintenance  budget  communications  include  the  transfer  of  all  budgetary 
information  required  to  create  a  common  situational  picture  to  facilitate  maintenance 
management. 


c.  Communicate  Maintenance  Reports 

Maintenance  report  communications  include  the  transfer  of  all  reports  available  to  the 
maintenance  management  teams  providing  the  required  maintenance  information  to  direct 
and  manage  resources.  Typical  maintenance  reports  may  include  but  are  not  limited  to 
progress,  logistical,  equipment  status,  and  budgetary  information. 

d.  Communicate  Maintenance  Requirements 

Maintenance  requirements  communications  include  the  transfer  of  all  information 
involving  established  maintenance  standards  and  procedures  approved  by  Technical 
Authority. 


e.  Communicate  Maintenance  Contracting  Requirements 

Maintenance  contracting  requirements  communications  includes  the  transfer  of  all 
contracts  related  information.  Effective  contracting  requirements  communication  should 
facilitate  the  process  of  placing  new  and  emergent  required  work  on  contract  for  execution. 
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3.  To  Organize 

The  function  ‘to  organize’  includes  the  available  CMMS  activities  that  provide 
structure  and  order  to  maintenance  management  information.  Effective  organization  provides 
a  common  structuring  system  to  the  data  available  in  the  program. 

a.  Organize  Execution  Plan 

Execution  planning  organization  includes  the  orchestration  of  maintenance  activities 
to  provide  an  integrated  execution  plan  that  reduces  delays  and  work  conflict. 

b.  Organize  Reports 

Reports  organization  includes  managing  the  systematic  generation  of  all  maintenance 
management  reports. 

c.  Organize  Maintenance  Teams 

Maintenance  team  organization  includes  the  orchestration  of  maintenance  teams  and 
other  available  personnel  to  properly  execute  an  integrated  execution  plan. 

d.  Organize  Schedule  Requirements 

Schedule  requirements  organization  includes  the  effective  generation  and  upkeep  of 
an  integrated  work  schedule  that  allows  for  all  required  work  to  be  executed  sequentially  and 
on  time  with  minimal  delays  and  work  conflict. 

e.  Organize  Maintenance  Requirements 

Maintenance  requirements  organization  includes  maintaining  the  current  and 
established  maintenance  requirements,  standards,  and  guidance  database  to  ensure  all 
executed  work  meets  the  necessary  quality  assurance  requirements. 

4.  To  Direct 

The  function  ‘to  direct’  includes  the  available  CMMS  activities  that  enable  the 
direction  of  action  required  in  maintenance  management. 
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a.  Direct  Maintenance  Execution 

Maintenance  execution  direction  includes  the  authorization  to  perform  work  in 
accordance  with  plans,  contracts,  and  schedules. 

b.  Direct  Maintenance  Planning 

Maintenance  planning  direction  includes  the  authorization  of  maintenance  activities 
to  plan  all  maintenance  that  shall  be  performed  in  a  specified  period. 

c.  Direct  Maintenance  Team 

Maintenance  team  direction  includes  maintenance  activities  authorizations  to  perform 
assigned  work  in  accordance  with  plans  and  schedules. 

d.  Direct  Maintenance  Contracting  Requirements 

Maintenance  contracts  direction  includes  directing  the  contractual  authorizations 
required  by  maintenance  activities  to  perform  work  in  accordance  with  plans  and  schedules. 

5.  To  Control 

The  function  ‘to  control’  includes  the  available  CMMS  activities  that  allow  managers 
to  exert  control  and  manage  certain  finite  aspects  of  maintenance. 

a.  Control  Maintenance  Budget 

Budget  maintenance  control  allows  the  managers  to  properly  fund  work-orders  while 
staying  within  the  budgetary  limits  and  constraints. 

b.  Control  Maintenance  Execution 

Maintenance  execution  control  allows  the  managers  to  properly  direct  work 
execution  in  accordance  with  schedule  and  contractual  limits  and  constraints. 

c.  Control  Maintenance  Requirements 

Maintenance  requirements  control  allows  the  managers  to  properly  execute  work  in 
accordance  with  approved  maintenance  requirements  and  standards. 
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C.  MAINTENANCE  MANAGEMENT  FUNCTIONAL  CYCLE  ANALYSIS 

The  work-order  lifecycle,  from  the  maintenance  manager’s  viewpoint,  consists  of 
four  stages:  validation,  planning,  execution,  and  closeout.  Each  stage  contains  several 
CMMS  maintenance  management  functions;  some  functions  remain  relevant  throughout 
several  stages. 

The  first  stage  is  work-candidate  validation,  screening,  and  brokering.  During  this 
stage  the  maintenance  managers  review  the  work-candidates  and  validate  them  to  determine 
the  request’s  reasonability.  Work-candidate  reasonability  is  primarily  determined  by  the 
equipment’s  degradation  impact  on  the  ship’s  mission.  Once  the  work  candidate  is  validated, 
the  manager  screens  it  by  determining  the  specific  maintenance  execution  period  and  brokers 
it  by  selecting  the  maintenance  activity  that  will  execute  the  work.  This  stage  requires  the 
functional  elements  used  for  preliminary  planning,  schedule  and  maintenance  requirements, 
and  communication. 

The  second  stage  is  work-order  planning.  During  this  stage  the  maintenance  manager 
and  maintenance  teams  determine  the  correct  technical  specifications,  cost  estimates,  overall 
planning,  and  best  practices  and  approach  for  the  specific  work-orders.  The  work-order 
planning  stage  employs  most  of  the  CMMS  functional  elements  and  involves  the  integrated 
coordination  of  the  maintenance  team. 

The  third  stage  is  work  execution.  All  the  planned  work  is  executed  during  this  stage 
in  accordance  with  the  work-orders  and  conforming  technical  authority.  Additionally,  any 
new  work  or  growth  work  that  arises  during  execution  begins  the  cycle  at  the  first  stage.  The 
execution  stage  employs  all  of  the  CMMS  functional  elements  as  it  involves  steps  from 
every  stage  of  the  maintenance  management  functional  cycle. 

The  fourth  stage  is  work  closeout.  During  work  closeout  all  the  maintenance 
activities  input  the  final  work-order  information  and  report  the  work  status.  If  neglected,  the 
lack  of  closeout  work-order  information  in  the  CMMS  hampers  the  CBM-i-  approach.  CBM-i- 
relies  on  up-to-date  equipment  status  and  performance  data  to  determine  trends  and  conduct 
RCM  analysis. 
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The  four  principal  work-order  lifecycle  stages  and  their  corresponding  maintenance 
management  functions  are  summarized  in  Table  3. 


Table  3.  Work-order  management  lifecycle  functional  allocation 


Work  Candidate 
Validation, 
Screening,  and 
Brokering 

Work  Order 
Planning 

Work  Order 
Execution 

Work  Order 
Closeout 

To  Plan 
Budgeting 
Logistics 
Schedulir\g 
Manpower 
requirements 

To  Plan 
Execution 
Budgeting 
Logistics 
Scheduling 
Manpower 
requirements 

To  Plan 
Execution 
Budgeting 
Logistics 
Scheduling 
Manpower 
requirements 

To  Plan 
Scheduling 

To  Communicate 
Planning 
Budget 

Reports 

Maintenance 

requirements 

Contracting 

requirements 

To  Communicate 
Planning 
Budget 

Reports 

Maintenance 

requirements 

Contracting 

requirements 

To  Communicate 
Planning 
Budget 

Reports 

Maintenance 

requirements 

Contracting 

requirements 

To  Communicate 
Budget 

Reports 

Maintenance 

requirements 

Contracting 

requirements 

To  Oreanize 
Reports 
Schedule 
requirements 

Maintenance 

requirements 

To  Organize 
Execution  Plan 
Reports 
Maintenance 

teams 

Schedule 

requirements 

Maintenance 

requirements 

To  Organize 
Execution  Plan 
Reports 
Maintenance 

teams 

Schedule 

requirements 

Maintenance 

requirements 

To  Oreanize 
Reports 
Maintenance 
requirements 

To  Direct 
Maintenance 

team 

To  Direct 
Planning 
Maintenance 

team 

Contracting 

requirements 

To  Direct 
Execution 
Planning 
Maintenance 

team 

Contracting 

requirements 

To  Direct 
Maintenance 

team 

Contracting 

requirements 

To  Control 
Requirements 

To  Control 
Budget 

Requirements 

To  Control 
Budget 
Execution 
Requirements 

To  Control 
Budget 
Execution 
Requirements 

D.  CMMS  REQUIREMENTS  DEFINITION 

The  requirements  definition  process’  purpose  “is  to  define  the  requirements  for  a 
system  that  can  provide  the  services  needed  by  users  and  other  stakeholders  in  a  defined 
environment”  (International  Organization  for  Standardization,  and  International 

Electrotechnical  Commission  2008,  36).  The  CMMS  needs  are  derived  from  conversations 
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with  CMMS  users,  maintenance  management  stakeholders,  and  DOD  and  NAVSEA  source 
documents.  These  needs  frame  the  MOE  development  process. 


1.  Stakeholders’  Needs 

A  stakeholders’  CMMS  needs  summary  is  displayed  below  in  table  4.  The  categories 
grouping  the  needs  are  related  to  the  aspect  of  the  system  addressing  the  individual  needs. 


Table  4.  Stakeholder  need  summary 


Interface  Needs 

Data  Generation  Needs 

Management 

Provide  a  simple  interface  to 
update  work-order  data 

Generate  work-orders  list 

Track  progress  and  deferred 
work-orders 

Allow  the  user  to  record  as  much 
work-order  data  as  possible 

Generate  a  work  schedule 

Allow  for  availability  planning 
and  scheduling  (IMS) 

Provide  dashboards  with  built  in 
analysis,  reasoning,  and  decision 
support  that  minimizes  user 
interaction 

Provide  maintenance  data  in 
support  of  reliability, 
maintainability,  and  sustainability 
analysis 

Facilitate  coordination  and 
sharing  of  data 

Be  scalable  to  different  classes  of 
hardware  and  software  platforms 
including  Navy  maintenance 
legacy  programs. 

Process  and  display,  as  part  of  a 
prognostic  or  diagnostic 
evaluation,  the  equipment  and 
system  health  to  the  crew 

Avoid  recurring  cost  licensing 

Be  operable  with  different  classes 
of  hardware  and  software 
platforms 

Collect  meter  data 

Reduce  planning  time 

Be  customizable  to  the  user 

Display  equipment  performance 
trends 

Increase  planning  accuracy 

Provide  a  back  door  for  flexible 
customization  by  IT  personnel 

Allow  extraction  of  ad  hoc 
reports 

Reduce  data  entry  errors 

Conform  to  all  applicable  Navy 
and  DOD  IT  policies  and 
requirements 

Provide  automated  sustainment 
info  and  metrics 

Reduce  time  delays  to  process 
work  authorization  and  planning 

Have  a  web  browser  based  UI  to 
allow  off  ship  support  to  view  the 
same  maintenance  info  as 
onboard  the  ship 

Provide  equipment  data  that  may 
be  transmitted  for  notification, 
analysis,  and  archival  purposes 

Optimize  TOC 

Provide  accurate  supply  data 

Provide  risk  predictions 

The  hierarchical  tree  in  Eigure  19  expands  on  the  three  categories  in  the  stakeholder  need 


summary  displayed  above  in  Table  4. 
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CMMS 

Needs 


Figure  19.  CMMS  needs  hierarchical  breakdown 
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2. 


CMMS  Constraints  Definitions 


The  CMMS  requirements  are  bounded  by  constraints  imposed  on  the  system.  The 
constraints  included  in  this  analysis  are  derived  from  the  limitations  encountered  by  the  LCS 
program. 

CMMS  communication  functionality  is  constrained  by  unreliable  ship-to-shore  CBM 
data  transmission  (United  States  Government  Accountability  Office  2013).  The  shipboard 
broadband  communication  capabilities  limit  the  ability  to  transmit  CBM+  data.  Real-time  or 
near-real-time  equipment  data  is  fundamental  for  the  successful  implementation  of  a  CBM-i- 
program.  Currently,  the  majority  of  the  equipment  data  is  inputed  manually  by  maintainers 
while  executing  work-orders  because  Navy  ships  are  not  designed  with  the  sensors  and 
meters  required  for  CBM-i-  data  recording  automatization. 

The  COTS  CMMS  program  currently  used  for  maintenance  management  does  not 
interface  with  all  Navy  maintenance  management  legacy  programs.  This  constraint  prevents 
the  development  of  a  common  maintenance  picture  and  results  in  the  use  of  ad-hoc 
reporting. 


3.  CMMS  Requirements 

The  following  requirements  are  derived  from  the  stakeholders’  CMMS  needs. 

•  The  CMMS  shall  provide  the  user  the  ability  to  record  and  update  all  critical, 
descriptive,  maintenance  data  on  a  work-order  as  required  by  the  specific 
equipment. 

•  The  CMMS  shall  display  a  customizable  dashboard  that  provides  decision 
makers  with  all  the  required  necessary  information  to  conduct  maintenance 
management  and  other  related  tasks. 

•  The  CMMS  shall  be  scalable  to  different  classes  of  hardware  and  software 
platforms. 

•  The  CMMS  shall  be  operable  with  different  classes  of  hardware  and  software 
platforms. 

•  The  CMMS  shall  be  customizable. 

•  The  CMMS  shall  possess  a  web  browser  user  interface  that  provides  the  user 
with  the  most  up-to-date  maintenance  data. 
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•  The  CMMS  shall  generate  reports  to  facilitate  analysis  of  equipment 
reliability,  maintainability,  sustainability,  consumption,  and  performance  as 
required  to  support  maintenance  management. 

•  The  CMMS  shall  generate  a  work-order  list  by  user  modified  query. 

•  The  CMMS  shall  generate  an  accurate  integrated  work-order  execution 

schedule. 

•  The  CMMS  shall  record  and  maintain  equipment  maintenance  data. 

•  The  CMMS  shall  provide  all  users  a  common  maintenance  status. 

•  The  CMMS  shall  allow  users  to  coordinate  and  share  maintenance 
requirements. 

•  The  CMMS  shall  maintain  work  execution  status. 

•  The  CMMS  should  not  require  periodic  licensing. 

E.  POSITED  SOLUTION  SET 

The  CBM-i-  approach  requires  a  CMMS  that  satisfies  the  stakeholder  requirements  in 
order  to  facilitate  maintenance  management.  As  the  aim  of  this  study  is  to  develop  a  set  of 
MOEs  applicable  to  any  CMMS,  it  is  assumed  that  the  selected  CMMS  can  potentially 
satisfy  all  stakeholder  requirements  if  used  correctly  and  effectively.  This  assumption 
provides  a  basis  for  logical  reasoning. 

A  satisfactory  CMMS  is  one  that  aids  decision  makers  to  manage  maintenance  by 
providing  the  functions  to  forecast,  plan,  communicate,  and  record  maintenance  history  as 
required. 

F.  THEORETICAL  FOUNDATIONS 

The  successful  employment  of  the  posited  solution  set  is  rooted  in  management 
theory,  organization  theory,  and  decision  theory.  These  theories  are  a  foundation  to  the 
understanding  of  management  and  decision  making  dynamics. 

I.  Management  Theory 

Management  is  defined  as  “designing,  providing,  and  maintaining  a  conducive 
internal  environment  in  tune  with  the  opportunities  and  challenges  of  the  external 
environment  through  planning,  organising,  directing,  and  controlling  all  resources  and 
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operations  in  order  to  achieve  effective  organisational  strategies  efficiently”  (Rao  2010,  3). 
The  theory  explains  management  as  a  discipline  composed  by  sets  of  functions,  roles,  skills, 
and  functional  areas  derived  through  management  thought  (Rao  2010,  3-16). 

Management  theory  provides  the  functional  capabilities  discussed  earlier  in  this 
chapter  (see  Figure  18).  The  theory  also  delineates  the  different  management  team  member 
roles  and  skills  required  for  effective  management. 

2.  Decision  Theory 

Decision  theory  explains  “how  individuals  or  groups  make  or  should  make  decisions” 
(Resnik  1987,  3).  A  ship  maintenance  management  decision  maker  expects  the  CMMS  to 
provide  reliable  equipment  data  to  determine  the  maintenance  requirements  based  on 
forecasting  and  planning.  Decision  theory  explains  how  an  individual  should  make  a 
decision  involving  “two  or  more  options  or  acts,  each  of  will  produce  one  of  several 
outcomes”  (Resnik  1987,  6).  For  example,  a  maintenance  manager  might  face  the  choice  of 
deciding  whether  to  execute  or  defer  maintenance  based  on  equipment  trends  and  reliability 
requirements.  The  decision  of  maintenance  execution  has  several  possible  outcomes  such  as: 
defer  and  the  equipment  does  not  fail,  defer  and  the  equipment  fails,  execute  early  and 
potentially  wasting  funding  on  unnecessary  maintenance,  execute  maintenance  on  time.  Each 
of  these  outcomes  has  a  set  of  consequences  that  affect  the  decision  maker’s  thought  process. 

Probability  theory  and  utility  theory  expand  the  aspects  of  the  decision  theory  that 
explain  how  individuals  or  groups  make  decisions  based  on  sets  of  known  probabilities  and 
conditions  of  different  states  of  a  problem.  Utility  theory  explains  how  individuals  or  groups 
make  decisions  in  order  to  maximize  their  utility.  Luce  and  Raiffa  (1957, 15)  describe  utility 
theory-based  decision  making  as:  “Given  a  set  of  possible  acts,  to  choose  one  (or  all)  of 
those  which  maximize  (or  minimize)  some  given  index”.  Measurable  utility  is  specific  to  the 
organization  conducting  the  maintenance  management.  Ship  maintenance  traditionally  seeks 
to  maximize  reliability  and  availability  while  minimizing  maintenance  costs  and  equipment 
downtime.  There  are  more  measurable  maintenance  utility  categories  but  traditionally  cost, 
reliability,  and  availability  are  the  two  that  lie  at  a  maintenance  organization’s  heart. 
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G. 


INTEGRATIVE  FRAMEWORK  FORMALIZATION 


The  integrative  framework  has  nine  cardinal  points  created  by  the  intersection  of  the 
problem’s  subjective  and  objective  domains.  This  research  includes  MOEs  developed  to 
fulfill  the  intersections  of  the  process  domain’s  ‘Cognitive’  and  ‘Procedures’  columns  with 
the  object  domain’s  “Object”  and  “Function”  rows. 

I.  Cognitive  Domain 

The  cognitive  domain  captures  the  abstractions  and  reasoning  that  arise  with  the 
CMMS  as  a  management  product  and  service.  The  domain  addresses  the  following 
relationships  cognition-object,  cognition-functions,  and  cognition-behavior. 

a.  Cognition-Object 

The  cognition-object  cardinal  point  seeks  to  identify,  interpret,  and  ascribe  meaning 
to  the  object  that  provides  the  service  (G.  O.  Langford  2012,  89).  The  object  is  the  CMMS 
program.  The  MOEs  developed  in  this  cardinal  point  relate  to  the  user’s  experiences  with  the 
CMMS  technology;  it  specifically  addresses  the  consequences  of  using  the  CMMS  and  the 
program’s  anticipated  responses  (G.  Langford,  Determinants  of  Deterrence  Effectiveness 
2014,  23). 

The  cognition-object  cardinal  point  takes  into  account  all  functional  aspects  of  the 
CMMS.  Some  questions  that  arise  in  determining  the  evaluation  factors  for  the  cognition- 
object  cardinal  point  are  (G.  O.  Langford  2012,  90-91): 

•  How  does  the  user  operate  the  CMMS? 

•  What  decisions  are  made  because  of  the  CMMS? 

•  How  is  the  CMMS  organized  for  use? 

•  How  is  the  CMMS  operated? 

•  What  type  of  feedback  does  the  CMMS  provide  the  user? 

•  How  does  the  CMMS  communicate  with  other  objects? 

(1)  MOE-e:  CMMS  Experience 

CMMS  users  may  execute  actions  related  to  the  five  maintenance  management 


functions:  planning,  organizing,  controlling,  directing,  and  communicating.  If  the  CMMS’ 
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purpose  is  to  assist  maintenance  activities  in  decision  making  regarding  planning, 
management,  and  administration  of  maintenance  actions,  then  the  effectiveness  of  the  system 
as  it  relates  to  the  user’s  experience  indicates  how  well  the  users  utilize  the  system’s 
available  capabilities  to  perform  each  of  the  maintenance  functions. 

The  CMMS  capability  employment  may  be  measured  by  determining  the  percentage 
of  work-orders  that  require  typical  management  tasks  to  happen  outside  of  the  CMMS.  To 
capture  this  information,  the  CMMS  IT  group  could  embed  a  task  breakdown  checklist 
within  the  work-order  for  the  user  to  indicate  which  typical  steps  related  to  that  task  at  hand 
are  being  performed  outside  of  the  system.  The  items  on  the  task  breakdown  checklist  would 
only  include  those  steps  that  are  expected  to  happen  within  the  system.  For  example,  in  order 
to  build  an  accurate  maintenance  history  it  is  important  that  tasks  such  as  ‘Indicate 
maintenance  completion’  or  ‘Provide  reason  for  work  deferral’  be  completed  and  captured  in 
the  system.  However,  a  common  administrative  task  like  contract  negotiation  should  not  be 
expected  to  take  place  within  the  CMMS.  Although  the  information  and  cost  data  resulting 
from  the  negotiations  belong  in  the  database,  it  might  be  unreasonable  to  expect  a 
government  contracting  officer  representative  and  a  contractor  to  conduct  cost  negotiations 
via  the  CMMS. 

The  CMMS  experience  MOE  (MOE-e)  provides  insight  into  the  effectiveness  of 
system  usage  as  it  relates  to  recording,  updating,  coordination,  and  communication  of 
maintenance  data. 

„  #  Task  Elements  Completed  ETsing  CMMS 

MOE-e  ^ 5 - 

#  Available  Task  Elements  Expected  to  be  Completed  ETsing  CMMS 

(2)  MOE-r:  CMMS  Response 

The  CMMS’  response  to  the  user  and  other  objects  is  a  system  characteristic  that 
expands  through  all  five  maintenance  management  functions.  CMMS  effectiveness  as  it 
relates  to  the  system’s  response  indicates  how  well  the  system  interacts  with  other  systems 
and  the  users.  System  feedback  through  the  user  interface  provides  an  alert  describing  the 
success  status  of  an  executed  action  within  the  system.  A  user’s  action  success  status  alert 
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may  indicate  the  accuracy  and  completeness  of  the  equipment  data  that  is  being  inputted  into 
the  system. 

Satisfactory  CMMS  responsiveness  may  be  measured  by  determining  the  percentage 
of  work-orders  requiring  additional  reviews  as  a  result  of  incorrect  or  incomplete  work-order 
data.  To  capture  this  information,  the  maintenance  management  team  should  capture  any 
instances  of  administrative  work-order  rework.  Any  type  of  rework  resulting  from  bad  or 
incomplete  data  could  be  considered  a  missed  opportunity  for  the  CMMS  to  provide 
feedback  by  alerting  the  user  of  the  error. 

The  CMMS  response  MOE  (MOE-r)  provides  insight  into  the  effectiveness  of  the 
management  and  reporting  of  maintenance  data  and  the  system’s  interface  with  objects. 

„  #  Work  Orders  Requiring  Administrative  Rework 

MOE-r  = - — - ^ - 

#Work  Orders 


b.  Cognition-Function 

The  cognition-function  cardinal  point  seeks  to  identify,  interpret,  and  ascribe 
meaning  to  the  CMMS  program’s  functions  (G.  O.  Eangford  2012,  89).  The  MOEs 
developed  in  this  cardinal  point  relate  to  the  predictions  and  expectations  of  events  resulting 
from  the  system’s  exchange  of  energy,  matter,  material  wealth,  and  information  (EMMI)  (G. 
Eangford,  Determinants  of  Deterrence  Effectiveness  2014,  23).  The  principal  focus  of  this 
MOE  development  analysis  is  on  the  ‘information’  element  of  EMMI  exchange. 

The  cognition-function  cardinal  point  involves  the  reasoning  of  the  CMMS’ 
functional  aspects.  Some  questions  that  arise  in  determining  the  cognition-function  cardinal 
point  evaluation  factors  are  (G.  O.  Eangford  2012,  90-91): 

•  How  will  the  user  enable  and  use  the  CMMS  functions? 

•  What  processes  are  necessary  to  assist  the  user  to  make  full  use  of  the 
CMMS? 

•  How  is  the  CMMS  organized  by  functions? 

•  How  does  the  CMMS  functional  organization  impact  user  behavior? 

•  How  are  functions  sequenced  in  the  CMMS  for  the  various  maintenance 
roles? 
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•  Are  the  CMMS  functions  being  performed  properly? 

(1)  MOE-g:  CMMS  Functionality  Prediction 

The  CMMS  serves  its  purpose  as  a  maintenance  management  tool  only  when  it 
exchanges  data  with  an  object.  The  ‘object’  exchanging  data  with  the  CMMS  in  a  CBM+ 
approach  could  be  a  meter  or  sensor  installed  on  monitored  equipment  or  the  user.  The 
CMMS  functionality  prediction  provides  an  object-system  information  exchange 
consequence  indication.  CMMS  effectiveness  as  related  to  the  system’s  functionality 
prediction  can  be  linked  to  how  well  it  provides  the  user  with  the  expected  tools  to  complete 
the  maintenance  management  tasks  for  which  they  are  responsible.  For  a  CMMS  program  to 
be  considered  fit-for-purpose  it  shall  provide  all  users  the  required  capabilities  and 
functionalities  to  be  able  to  properly  manage  a  work-order  in  accordance  with  regulations. 

Functionality  prediction  may  be  measured  by  periodic  evaluation  of  the  CMMS. 
These  evaluations  should  be  completed  by  CMMS  trained  users  who  perform  tasks  requiring 
data  input  or  review  of  the  data  that  has  been  entered  into  the  system.  The  evaluation  shall  be 
customized  for  the  user-evaluator  to  include  categories  applicable  only  to  the  evaluator’s 
area  of  responsibility  within  the  maintenance  management  cycle.  The  functionality 
categories  include  the  necessary  processes  to  complete  maintenance  tasks  and  system 
organization.  Users  can  evaluate  the  system  on  how  well  it  follows  the  maintenance  process 
once  information  has  been  entered.  For  example,  if  a  sensor  indicates  a  reduction  in  oil  flow 
in  a  main  reduction  gear  nozzle,  a  user  should  predict  the  receipt  of  an  alert  when  the  CMMS 
receives  the  oil  flow  data  from  the  sensor.  A  different  example  could  be  a  maintainer’s 
prediction  that  after  entering  a  new  work-order  request  in  the  system  it  will  alert  the  port 
engineer  for  revision  and  approval.  In  both  of  these  cases  the  users  prognosticate  the  CMMS  ’ 
function  predictability.  A  predictable  system  enables  all  users  to  trust  the  system. 

Once  the  user  scores  are  compiled,  they  shall  be  weighed  and  normalized.  The 
resulting  ‘CMMS  functionality  score’  provides  an  estimate  of  how  well  the  program 
performs  the  predicted  traditional  steps  required  for  a  user  to  complete  their  maintenance 
related  tasks  while  meeting  their  expectation  as  it  relates  to  CMMS  functionality.  CMMS 
expectations  should  be  formed  and  evaluated  by  users  with  CMMS  training.  These  users  will 
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understand  the  attainable  functionality  of  the  system  and  how  it  differs  from  other 
maintenance  databases  and  programs. 

The  CMMS  functionality  expectation  MOE  (MOE-g)  provides  insight  into  the 
effectiveness  of  the  predicted  functional  results  arising  from  interactions  between  the 
systems  and  data  reporting  objects. 

n 

MOE-g  =  F.C.i  xlTj  -fF.C.2  x1E2...  +  F.C.„  xW^  =  Y^F.C.^W. 

i=l 

Where  F.C.i  =  F.C.i  F.C.2, ...  F.C.n  are  the  functionality  category  scores  assigned  during  the 
evaluation  and  Wi  =  W2, ...  Wn  are  the  weights  associated  with  each  category.  The  category 

weights  may  be  assigned  using  a  multiple  criteria  analysis  weighing  process. 

(2)  MOE-c:  CMMS  Eunctionality  Expectation 

The  CMMS  functionality  expectation  MOE  provides  a  measurement  of  the  users’ 
perceived  system  fitness-for-purpose  based  on  how  well  the  CMMS  meets  the  expected 
results.  Result  expectations  are  based  on  the  user’s  desired  system  functional  requirements. 
Eor  a  CMMS  program  to  be  effective  it  shall  provide  users  a  set  of  expected  results  and 
capabilities  once  the  maintenance  data  has  been  entered. 

This  MOE  is  measured  with  the  same  method  used  to  calculate  MOE-g,  functionality 
prediction.  This  method  requires  a  periodic  evaluation  of  the  CMMS  functionality  by 
selected  users.  Users  can  evaluate  the  system  on  how  well  its  functionality  and  their 
expectations  in  accomplishing  maintenance  tasks  match.  Eor  example,  a  maintenance 
manager  who  is  planning  the  work  package  for  upcoming  maintenance  availabilities  should 
expect  the  CMMS  to  generate  a  preliminary  list  of  potential  work-orders  based  on  equipment 
maintenance  history.  If  the  CMMS  does  not  generate  the  work-order  candidate  list  then  the 
user’s  expectations  are  not  met  and  the  system’s  functionality  expectations  are  evaluated 
accordingly.  Another  possibility  is  that  the  program  only  generates  a  list  of  scheduled  or 
overdue  preventive  maintenance  work-orders  but  fails  to  recommend  new  work-orders  based 
on  equipment  condition  assessments  and  findings.  If  the  maintenance  manager’s  expectation 
is  that  a  satisfactory  CMMS  should  automatically  recommend  work  based  on  condition 
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assessments  then  the  evaluation  should  result  in  a  MOE  that  indicates  partial  meeting  of 
needs  and  an  opportunity  to  improve  the  system’s  functionality. 

The  CMMS  functionality  expectation  MOE  (MOE-c)  identifies  user  satisfaction 
based  on  system  expectations.  A  low  score  could  potentially  indicate  an  opportunity  for 
system  or  process  improvement. 

n 

MOE-c  =  F.C.E.,  xW  +F.C.E.,  xW,...  + F.C.E.^  xW  ^Yf.C.E.W. 

11  2  2  n  n  i  i 

i=\ 

where  F.C.E.i  =  F.C.E.j^  F.C.E.2,  ...  E.C.E.n  are  the  expectation  score  assigned  during  the 
evaluation  for  the  specific  functionality  category  and  IT,  =  IT;.  W2, ...  W„  are  the  weights 
associated  with  each  category. 

c.  Cognition-User 

The  cognition-user  cardinal  point  seeks  to  identify,  interpret,  and  ascribe  meaning  to 
the  CMMS  program’s  user  behaviors  when  the  program  is  in  use  and  when  it  is  not  available 
or  in  use  (G.  Eangford,  Determinants  of  Deterrence  Effectiveness  2014,  23). 

The  cognition-user  cardinal  point  involves  the  reasoning  of  the  CMMS  user’s 
behaviors  before  the  program  is  used,  during  its  use,  and  after  its  use  and  their  anticipation 
when  it  is  not  in  use  (G.  O.  Eangford  2012, 90).  Some  questions  that  arise  in  determining  the 
cognition-user  cardinal  point  evaluation  factors  are  (G.  O.  Eangford  2012,  90-91): 

•  What  should  the  users  anticipate  when  the  CMMS  is  in  use? 

•  What  should  the  users  anticipate  when  the  CMMS  is  unavailable? 

•  What  should  the  users  anticipate  when  the  CMMS  is  not  in  use? 

•  How  do  users  organize  their  work  with  and  without  the  CMMS? 

•  How  are  the  users’  behaviors  changed  to  complete  the  maintenance 
management  tasks  based  on  the  CMMS  limitations? 

Cognition-user  MOEs  are  not  developed  in  this  research.  These  MOEs  are  better 
developed  once  a  CMMS  has  been  selected  and  implemented  and  is  being  operated  by  all  the 
maintenance  management  users.  The  organization  members  determined  to  be  users  will 
shape  this  cardinal  point. 
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2. 


Procedural  Domain 


The  procedural  domain  captures  the  procedures  and  mechanisms  of  the  CMMS  as  a 
management  product  and  service.  The  domain  addresses  the  following  relationships 
procedure-object,  procedure  -functions,  and  procedure  -behavior. 

a.  Procedure-Object 

The  procedure-object  cardinal  point  seeks  to  identify  the  validity  of  the  CMMS 
selection  and  development  of  CMMS  procedures  (G.  Langford,  Building  the  Determinants  of 
Technology  Effectiveness  2014, 12).  The  MOEs  developed  in  this  cardinal  point  relate  to  the 
effectiveness  of  the  specific  CMMS  selection  and  to  how  the  technical  aspects  of  the 
system’s  processes  meet  the  stakeholder’s  needs  (G.  Eangford,  Determinants  of  Deterrence 
Effectiveness  2014,  23). 

The  procedure-object  cardinal  point  focuses  on  the  processes  and  mechanisms  related 
to  the  CMMS.  Some  questions  that  arise  in  determining  the  evaluation  factors  for  the 
procedure-object  cardinal  point  are  (G.  O.  Eangford  2012,  91): 

•  Do  the  CMMS  procedures  reflect  the  maintenance  desired  outcome? 

•  Do  the  CMMS  procedures  reflect  the  project  requirements? 

•  Are  the  CMMS  procedures  driving  the  maintenance  management  towards  the 
desired  outcome? 

•  Does  the  CMMS  operate  as  desired  while  interacting  with  external 
influences? 

•  Does  the  CMMS  clearly  differ  from  other  competitive  products? 

(1)  MOE-s:  CMMS  Selection  Validity 

In  a  CBM-i-  environment  the  CMMS  is  not  effective  as  a  standalone  system.  The 
information  exchange  between  the  CMMS  and  the  data  providers  determines  the  system’s 
fitness-for-purpose.  The  CMMS’  goal  in  a  CBM-i-  management  approach  is  to  provide  the 
necessary  tools  for  the  maintenance  managers  to  execute  their  tasks  and  make  decisions  with 
the  greatest  amount  of  near  real-time  data.  Evaluating  the  CMMS’  validity  should  provide 
indications  of  the  program’s  suitableness  to  help  meet  the  overarching  maintenance 
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organization’s  goals  of  reduced  maintenance  costs  and  increased  equipment  life  expectancy 
and  availability. 

There  are  many  capable  and  competitive  CMMS  and  EAM  COTS  products  that  may 
be  selected  by  a  maintenance  organization.  The  ‘CMMS  selection  validity’  MOE  (MOE-s) 
evaluates  the  effectiveness  of  the  current  CMMS’  selection  as  it  relates  to  organizational 
goals.  The  validity  of  a  selected  CMMS  may  be  measured  by  the  periodic  evaluation  of  the 
system  as  an  object.  This  evaluation  should  be  performed  by  high  level  managers  and  power 
users  because  its  foci  are  bounded  by  high  level  organizational  goals.  The  evaluators  shall 
score  the  system  on  how  well  it  serves  as  a  tool  in  reducing  lifecycle  costs  and  increasing 
equipment  availability. 

The  CMMS  selection  validity  MOE  identifies  the  high  level  stakeholder’s 
satisfaction  with  the  selected  CMMS  based  on  programmatic  goals.  A  low  score  could 
potentially  indicate  that  the  CMMS,  as  an  object,  lacks  the  necessary  tools  to  enable  the 
attainment  of  the  programmatic  goals.  System  factor  categories  are  determined  by  the 
organization’s  stakeholders  and  may  include  categories  such  as  overall  availability,  cost 
control,  and  meeting  expected  equipment  operational  life. 

n 

MOE-s  =  5.F.j  xWi  +5.F.2  xW^...  +  S.F.^  xlf;  =  ^S.F..W. 

i=i 

Where  S.F.i  =  S.F.j^  S.F.2, ...  S.F.n  are  the  system  factor  scores  assigned  during  the  evaluation 
for  the  specific  system  factor  category  and  W,  =  W;.  W2, ...  Wn  are  the  weights  associated  with 
each  category. 

(2)  MOE-x:  CMMS  Operational  Context  Validity 

The  CMMS’  operation  is  influenced  by  external  objects.  Some  objects  that  influence 
the  CMMS  are  users,  hardware,  and  other  peripheral  equipment  and  software.  The  CMMS’ 
ability  to  interact  with  external  objects  determines  its  scalability,  flexibility,  and  adaptability. 
A  suitable  CMMS  should  require  minimal  effort  to  accomplish  interaction  with  different 
external  systems.  Evaluating  the  CMMS’  operational  context  validity  should  provide 
indications  of  the  program’s  ability  to  work  with  different  types  of  hardware  and  software. 
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The  maintenance  organization  benefits  from  a  scalable,  flexible,  and  adaptable  CMMS 
because  its  operation  can  potentially  be  expanded  to  new  platforms  such  as  handhelds  or 
other  organization’s  software. 

The  proper  measurement  of  any  software’s  scalability,  flexibility,  and  adaptability 
requires  a  set  of  software  engineering  skills  and  knowledge  not  traditionally  found  at  a  Navy 
Program  Office  or  maintenance  organization.  For  this  reason,  the  calculation  of  the  ‘CMMS 
operational  context  validity’  MOE  (MOE-x)  only  requires  input  that  can  be  provided  by  the 
CMMS  power  users.  Although  the  CMMS  power  users  and  maintenance  managers  may  not 
know  how  to  measure  a  program’s  scalability,  flexibility,  and  adaptability  they  should  be 
able  to  recognize  successful  and  unsuccessful  opportunities  when  the  CMMS  tries  to  interact 
with  other  systems.  The  validity  of  the  CMMS’  operational  context  may  be  measured  by 
combining  the  percentage  of  systems  that  successfully  interact  with  the  CMMS  and  the 
periodic  evaluation  of  the  system’s  interactions  with  external  objects  and  environments.  This 
evaluation  should  be  performed  by  the  CMMS  power  users  and  IT  personnel  with  input  from 
users  and  maintenance  managers. 


MOE-x  identifies  the  CMMS’  ability  to  interact  with  external  objects.  A  low  score 
could  potentially  indicate  limitations  in  the  CMMS’  capabilities  to  communicate  and  interact 
with  other  systems.  A  lack  of  communication  and  interaction  between  systems  results  in  a 
situation  that  requires  ad-hoc  actions  that  waste  time  and  resources.  Operational  context 
factors  are  determined  by  the  power  users  and  organization’s  stakeholders  and  includes 
categories  that  capture  the  evaluator’s  perceived  system’s  ability  to  interact  with  other 
systems.  The  perception  will  be  mostly  based  on  the  user’s  operational  experience  with  the 
CMMS. 


MOE-x  = 


MOE-x  = 


#  Systems  CMMS  Interacts  With 

#  Maintenance  Management  Systems  Available 

#  Systems  CMMS  Interacts  With 

#  Maintenance  Management  Systems  Available 


X  + o.c.i  X  Wj + ac.2  X  W2... + ac.„  x  w„ 

n 

i=\ 


Where  O.  C.  i  =  O.  C.  1,  O.  C.2, ...  O.  C. „  are  the  system  operational  context  factor  scores  assigned 
during  the  evaluation  for  the  specific  category  and  W,  =  W;.  W2,  ...  are  the  weights 
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associated  with  each  category.  W^is  the  weight  assigned  to  the  percentage  of  systems  that  are 
able  to  successfully  interact  with  the  CMMS. 

b.  Procedure-Function 

The  procedure-function  cardinal  point  seeks  to  define  and  identify  the  validity  of  the 
CMMS  resource  selection  and  the  development  of  procedures  (G.  Langford,  Building  the 
Determinants  of  Technology  Effectiveness  2014, 12).  The  MOEs  developed  in  this  cardinal 
point  relate  to  the  CMMS  functions’  representative  mechanisms  (G.  O.  Eangford  2012,  89). 
The  functions’  mechanisms  and  procedures  effectiveness  may  indicate  how  well  they  fit  the 
stakeholders’  organizational,  managerial,  and  maintenance  goals.  Effective  procedures  and 
mechanisms  result  in  less  waste  for  the  organization.  An  effective  CMMS  operates  with 
functions  that  support  these  procedures. 

Some  questions  that  arise  in  determining  the  evaluation  factors  for  the  procedure- 
function  cardinal  point  are  (G.  O.  Eangford  2012,  91): 

•  Are  the  CMMS  functions  constraining  resources  by  duplicating  efforts? 

•  Do  the  CMMS  functions  help  achieve  and  support  good  administrative 
practices  that  take  into  account  role  assignments  and  responsibility 
assignments? 

•  Do  the  CMMS  functions  reflect  desired  enterprise  metrics  and  project 
specifics? 

•  Do  the  CMMS  functions  interact  with  objects  outside  its  boundaries  as 
required  to  accomplish  the  management  tasks? 

(1)  MOE-u:  CMMS  Eunctional  Resource  Utilization  Validity 

Each  function  performed  by  the  CMMS  draws  from  a  finite  resource  pool.  These 
resources  may  be  monetary,  of  intellectual  capital,  or  personnel  related.  The  maintenance 
organization  depends  on  the  effective  management  of  these  resources  to  successfully 
accomplish  their  mission.  If  the  CMMS’  functions  and  organization  results  in  duplication  of 
efforts  then  it  creates  waste  in  the  form  of  time,  motion,  and  possible  talent  underutilization. 
A  suitable  CMMS  should  have  functions  that  support  good  administrative  practices  that 
minimize  any  type  of  waste. 
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Effective  resource  utilization  is  fundamental  to  successful  management.  Navy  surface 
maintenance  management  resources  are  additionally  constrained  by  the  fiscal  environment. 
A  CMMS  that  cannot  support  effective  resource  management  will  likely  fail  to  meet  the 
overarching  CBM+  organizational  need  to  reduce  total  ownership  cost.  Organizational 
resources  like  intellectual  capital  may  be  linked  to  performance  drivers  (Alcaniz,  Gomez- 
Bezares  and  Roslender  2011).  If  performance  drivers  are  linked  to  resource  management 
then  the  effective  administration  of  these  resources  enables  the  organization’s  maintenance 
managers  to  create  value  (Harazin  and  Padar  2013,  38).  The  ‘CMMS  functional  resource 
utilization  validity’  MOE  (MOE-u)  evaluates  the  system’s  functions  capacity  to  support 
effective  resource  management. 

Periodic  project  or  group  performance  evaluations  could  provide  insight  into  the 
efficiency  of  the  CMMS’  functions  to  support  resource  management.  If  effective  resource 
management  is  linked  to  performance  indicators  then  the  performance  indicators  should 
point  out  opportunities  to  improve  aspect  of  the  CMMS  functions  that  do  not  utilize  the 
available  organizational  resources  efficiently.  A  periodic  evaluation  of  the  system  by  CMMS 
users  using  a  balanced  scorecard  allows  them  to  assign  scores  to  the  system’s  functions  as 
they  relate  to  the  processes  individual  to  their  task.  It  is  important  that  the  system  evaluators 
understand  all  the  individual  tasks  that  are  performed  throughout  the  typical  CMMS  work- 
order  process.  Completing  a  value  stream  map  helps  users  understand  the  full  process  and 
facilitates  the  ability  to  recognize  steps  and  functions  that  create  waste. 

MOE-u  provides  a  metric  that  captures  the  system’s  aggregate  functional  resource 
utilization  score.  A  low  MOE  score  indicates  opportunities  for  process  improvement  by 
addressing  the  system’s  functions.  However,  it  is  not  necessary  to  wait  for  an  overall  low 
MOE-u  score  to  identify  opportunities  to  improve  the  process  as  they  relate  to  the  system’s 
functions.  The  organization  may  take  action  on  any  of  the  average  individual  functional 
balanced  scorecard  categories. 

n 

MOE-u  =  F.5.1  xffi;  +  F.5.2  +  xB;  = 

i=l 
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Where  F.S.i  =  F.S.]^  F.S.2, ...  F.S.n  are  the  average  individual  functional  balanced  scorecard 
scores  and  Wi  =  W;.  W2, ...  Wn  are  the  weights  associated  with  each  category. 

(2)  MOE-b:  CMMS  Functional  Boundary  Conditions 

Traditional  CMMS  programs  interact  with  other  objects  through  the  system’s 
functions.  These  objects  may  be  users,  other  systems,  and  peripherals.  A  boundary  condition 
is  the  “mediation  of  capabilities  that  enact  across  boundaries”  between  two  objects  (G.  O. 
Langford  2012, 43).  These  conditions  seek  to  describe  the  interactions  between  objects  that 
exist  within  their  respective  boundaries  (G.  O.  Langford  2012,  43).  A  suitable  CMMS’ 
functions  should  flawlessly  interact  with  other  objects  and  systems  to  accomplish 
maintenance  management  tasks. 

The  boundary  conditions  between  a  CMMS’  functions  and  other  objects  dictate  how 
well  the  systems’  operation  fits  its  intended  purpose.  Effective  boundary  conditions  enable 
the  transfer  of  EMMI,  specifically  energy  and  information,  between  systems.  Functional 
boundary  condition  effectiveness  may  be  described  by  characteristics  including  user 
interaction,  and  the  CMMS’  ability  to  transfer  data  between  systems  without  ad-hoc  data 
exchange  methods. 

MOE-b  provides  a  metric  of  the  CMMS’  function’s  ability  to  interact  with  external 
objects.  This  MOE  may  help  identify  limitations  in  the  CMMS’  functions’  capabilities  to 
communicate  and  interact  with  other  objects.  The  CMMS  boundaries  are  defined  by  the 
program,  the  power  users,  and  organization’s  stakeholders  and  includes  categories  that  can 
capture  the  evaluator’s  perceived  system’s  ability  to  interact  with  other  systems  using  a 
balanced  scorecard.  The  scores  are  based  on  the  evaluator’s  operational  experience  with  the 
CMMS’  functions. 


MOE-b  =  B.C.i  X  +  B.C.2  X  W2...  +  B.C.„  x 

i=\ 

Where  B.C ,  =  B.C  ■  1,  B.C  -i, ...  B.C  -n  are  the  average  individual  functional  boundary 
conditions  balanced  scorecard  scores  and  Wi  =  W2, ...  Wn  are  the  weights  associated  with 

each  category.  MOE-b  provides  a  metric  that  captures  the  system’s  aggregate  functional 
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boundary  conditions  score.  A  low  MOE  score  indicates  opportunities  for  improvement  by 
addressing  the  interaction  between  the  system’s  functions  and  external  systems. 

c.  Procedure-User 

The  procedure-user  cardinal  point  addresses  the  process  and  mechanisms  that 
describe  user  behaviors  due  to  the  CMMS  (G.  O.  Langford  2012,  90).  The  cardinal  point 
involves  the  influence  of  CMMS  procedures  on  user  behavior  (G.  Langford,  Determinants  of 
Deterrence  Effectiveness  2014, 23).  Some  questions  that  arise  in  determining  the  procedure- 
user  cardinal  point  evaluation  factors  are  (G.  O.  Langford  2012,  90-91): 

•  Do  the  CMMS  procedures  enable  good  sense  decisions? 

•  Do  the  CMMS  procedures  enable  the  production  of  credible  work? 

•  Do  the  CMMS  procedures  show  resilience  to  change? 

•  Are  the  CMMS  procedures  novel? 

Procedure-user  MOEs  are  not  developed  in  this  research.  These  MOEs  are  better 
developed  once  a  CMMS  has  been  selected  and  implemented  and  is  being  operated  by  all  the 
maintenance  management  users.  The  organization  members  determined  to  be  users  will 
shape  this  cardinal  point. 

3.  Model  and  Representation  Domain 

The  model  and  representation  domain  ascribes  meaning  to  the  CMMS  by  evaluating 
models  and  representations  of  the  system,  its  functions,  and  the  users  expectations  based  on 
these  models.  The  model  and  representation  domain  MOEs  are  not  developed  in  this 
research.  These  MOEs  are  better  developed  once  a  CMMS  has  been  selected  and 
implemented  within  an  organization.  The  other  systems  and  available  peripherals  will  shape 
this  domain. 

H.  MOE  TO  NEEDS  TRACEABILITY 

The  modified  nine  step  MOE  development  methodology  results  in  a  set  of  eight 
MOEs  that  address  the  Cognition- Object,  Cognition-Eunction,  Procedure-Object,  and 
Procedure-Eunction  cardinal  points  of  the  integrative  framework.  Each  developed  MOE 
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measures  a  CMMS  aspect  that  is  traceable  to  the  system  requirements  derived  from  the 
stakeholder  needs.  A  traceability  matrix  tracing  the  developed  MOEs  to  the  original  set  of 
stakeholder  needs  is  illustrated  in  Table  5.  The  system  requirements  are  summarized  in  Table 
6  and  the  eight  developed  MOEs  are  summarized  in  Table  7. 


Table  5.  Traceability  matrix 


Stakeholder  Requirements 

1.0 

2.0 

3.0 

4.0 

5.0 

6.0 

7.0 

8.0 

9.0 

10.0 

11.0 

12.0 

13.0 

Measures  of  Effectiveness 

MOE-e 

MOE-r 

MOE-g 

MOE-c 

MOE-s 

MOE-x 

MOE-u 

MOE-b 

The  traceability  matrix  in  Table  5  illustrates  that  each  stakeholder  requirement  that 
may  be  used  in  determining  the  CMMS’  fitness-for-purpose  is  addressed  by  at  least  one 
MOE  from  the  developed  set. 


Table  6.  CMMS  requirements  summary 


1.0 

The  CMMS  shall  display  a  customizable  dashboard  that  provides  decision  makers 
with  all  the  required  necessary  information  to  conduct  maintenance  management 
and  other  related  tasks. 

2.0 

The  CMMS  shall  be  scalable  to  different  classes  of  hardware  and  software 
platforms. 

3.0 

The  CMMS  shall  be  operable  with  different  classes  of  hardware  and  software 
platforms. 

4.0 

The  CMMS  shall  be  customizable. 

5.0 

The  CMMS  shall  possess  a  web  browser  user  interface  that  provides  the  user  with 
the  most  up-to-date  maintenance  data. 

6.0 

The  CMMS  shall  generate  reports  to  facilitate  analysis  of  equipment  reliability, 
maintainability,  sustainability,  consumption,  and  performance  as  required  to  support 
maintenance  management. 

7.0 

The  CMMS  shall  generate  a  work-order  list  by  user  modified  query. 

8.0 

The  CMMS  shall  generate  an  accurate  integrated  work-order  execution  schedule. 

9.0 

The  CMMS  shall  record  and  maintain  equipment  maintenance  data. 

10.0 

The  CMMS  shall  provide  all  users  a  common  maintenance  status. 

11.0 

The  CMMS  shall  allow  users  to  coordinate  and  share  maintenance  requirements. 

12.0 

The  CMMS  shall  maintain  work  execution  status. 

13.0 

The  CMMS  should  not  require  periodic  licensing. 
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Table  7.  MOE  summary 


MOE 

Description 

Calculation 

MOE-e:  CMMS 
experience 

Cognition-Object:  Provides  insight  into  the  effectiveness  of 
system  usage  as  it  relates  to  recording,  updating, 
coordination,  and  communication  of  maintenance  data. 

#  Tadc  Elements  Con^ileted  Using  CMMS 
#  Available  Task  Elements  Expected  to  be  Conqileted  Using  CMMS 

MOE-r:  CMMS 

Response 

Cognition-Object:  Provides  insight  into  the  effectiveness  of 
the  management  and  reporting  of  maintenance  data  and  the 
system’s  interface  with  objects. 

#  Work  Orders  Requiring  Administrative  Rew’ork 
#Work  Orders 

MOE-g:  CMMS 

Functionality 

Prediction 

Cognition-Function:  Provides  insight  into  the  effectiveness 
of  the  predicted  functional  results  arising  from  interactions 
between  the  systems  and  data  reporting  objects. 

F.C.xW,+F.C.,xW,...  +  F.C.  xW  =yF.C.W. 

1  1  z  z  n  n  i  1 

MOE-c:  CMMS 

Functionality 

Expectation 

Cognition-Function:  Identifies  user  satisfactionbasedon 
system  expectations. 

FCi:.,xW^  +F.CF.^xW^...  +  F.C.E.^xW^  =^F.C.E.^W. 

i=\ 

MOE-s:CMMS 

SelectionValidity 

Procedure-Object:  Identifies  the  high  level  stakeholder’s 
satisfaction  based  on  programmatic  goals . 

S.F.iX  fTi-f  5.F.2  xFr2...+  5.F.„xr„  =  ^  S.F.,W. 

1=1 

MOE-x:  CMMS 

Operational 

ContextValidity 

Procedure-Object :  Identifies  the  CMMS  ’  ability  to  interact 
with  external  objects. 

O.C.,  xW.+  O.C..  X  W,...  +  O.C.„  xW=y  O.C.W, 

11  z  t.  n  n  I  i 

j=l 

MOE-u:  CMMS 

Functional 

Resource 

Utilization  V  alidity 

Procedure-Function:  Provides  insight  into  the  efficiency  of 
the  CMMS  ’  functions  to  support  resource  management. 

F.S.xW,+F.S.,xW,...  +  F.S.  xW  =  yF.S..W. 

11  Z  Z  n  n  i  i 

i=l 

MOE-b:CMMS 

Functional 

Boundary 

Conditions 

Procedure-Fimction:  Provides  a  metric  of  the  CMMS  ’ 
function’s  ability  to  interact  with  external  objects. 

n 

B.C.,xW,  +B.C.,xW,...  +  B.C.xW=yB.C.W, 

11  Z  Z  n  n  1  1 

/=1 
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VI.  MEASURES  OF  EFFECTIVENESS  SET  DEVELOPMENT 
FINDINGS,  LIMITATIONS,  AND  CONCLUSIONS 

A.  MAJOR  FINDINGS 

The  nine-step  methodology  provides  a  repeatable  process  to  successfully  develop  a 
set  of  measures  of  effectiveness  (MOE)  based  on  stakeholder  needs,  requirements,  and 
system  limitations  to  determine  the  fitness-for-purpose  of  a  computerized  maintenance 
management  system  (CMMS)  and  its  employment. 

The  effective  implementation  of  a  Conditioned  Based  Maintenance  Plus  (CBM-i-) 
strategy  relies  on  the  effective  use  of  a  suitable  CMMS  program.  The  lack  of  well-defined 
and  methodically  derived  MOEs  is  a  consequence  of  the  absence  of  a  consistent 
methodology  to  measure  CMMS  implementation  effectiveness  as  it  pertains  to  CBM-i-.  Well- 
defined  MOEs  help  determine  and  monitor  how  well  the  selected  CMMS  program 
accomplishes  its  intended  operational  objectives.  Currently  there  is  no  standard  method  to 
develop  MOE  sets  based  on  objective  values  and  subjective  criteria.  A  well-developed 
CMMS  MOE  set  may  be  used  by  a  maintenance  organization  in  evaluating  the  CMMS’ 
fitness-for-purpose  throughout  its  lifecycle. 

A  set  of  CMMS  MOEs  was  developed  in  Chapter  V  using  a  modification  of 
Langford’s  nine-step  methodology  (G.  Langford,  Determinants  of  Deterrence  Effectiveness 
2014, 7-1 1).  The  nine-step  method  is  a  systems  engineering  (SE)  focused  approach  that  uses 
an  integrative  framework  of  objective  values  and  subjective  criteria  to  guide  the  development 
of  MOEs. 

The  method  consists  of  a  repeatable  process  and  allows  for  iterations  based  on 
requirements  derived  from  stakeholder  needs.  The  nine-step  method’s  repeatability  allows 
the  maintenance  organization  to  periodically  determine  the  effectiveness  of  their  CMMS 
employment.  Every  periodic  evaluation  should  indicate  opportunities  for  the  organization’s 
personnel,  and  maintenance  processes  and  mechanisms  to  improve. 
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B.  SIGNIFICANCE  OF  MAJOR  FINDINGS 

A  maintenance  organization  may  use  the  developed  set  of  eight  CMMS  MOEs  to 
determine  how  well  the  CMMS  is  being  employed  within  the  organization  and  how  well  it 
supports  a  CBM+  approach.  The  MOE  set  provides  basic  indicators  to  determine  the 
effectiveness  of  the  CMMS  function’s  and  of  the  system  as  an  object.  The  developed  MOEs 
also  address  the  initial  set  of  stakeholder  requirements  and  needs. 

Maintenance  organizations  can  use  the  development  processes  established  by  the 
nine-step  methodology  to  develop  valid,  significant,  and  useful  MOEs  for  system’s  fitness- 
for-purpose  evaluation  and  determination.  The  nine-step  method  is  practicable  within  a 
typical  surface  maintenance  organization.  Developing  and  iterating  MOEs  will  only  require 
staff  with  an  understanding  of  the  overarching  programmatic  goals  and  knowledge  of  the 
evaluated  systems  due  to  the  method’s  simple  and  direct  approach.  The  process  could  also  be 
extended  to  MOE  development  in  other  areas. 

The  developed  CMMS  MOEs  have  the  characteristics  specified  in  the  works 
reviewed  in  Chapter  III. 

The  works  of  Roedler  and  Jones  (2005, 36)  and  Stevens  (1986, 55)  describe  some  of 
the  traditionally  accepted  MOE  characteristics,  such  as: 

•  Provide  insight  into  at  least  one  requirement 

•  Provide  insight  into  different  aspects  of  the  alternative 

•  Shall  not  be  predefined 

•  Should  be  relevant 

•  The  set  should  be  complete 

•  Should  be  precisely  defined 

•  Should  be  expressed  in  terms  that  are  meaningful  to  testers  and  developers 

•  Meaning  should  not  be  open  to  interpretation  with  the  passage  of  time 

•  Inputs  should  be  measureable 

•  Measurements  should  not  interfere  with  system  operation 

•  All  qualitative  measurements  should  use  the  same  standard. 
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The  nine-step  methodology  also  enables  the  development  of  an  MOE  set  that 
addresses  the  five  different  organizational  levels  in  Wireman’s  performance  pyramid  (2005, 
220).  The  five  levels  are:  corporate,  financial,  tactical,  functional,  and  their  link  to  efficiency 
and  effectiveness. 

C.  LIMITATIONS 

Due  to  the  scope  of  this  research,  the  nine-step  method  was  modified  to  include  only 
the  first  seven  steps.  The  successful  completion  of  ‘Step  7:  Formalize  Integrative 
Framework’  produces  an  initial  set  of  MOEs  based  on  requirements,  needs,  and  the  system’s 
functional  limitations.  Steps  eight  and  nine  address  the  implementation  of  the  developed 
MOEs  and  are  more  applicable  for  an  organization  that  is  performing  the  developed 
measurements  to  determine  the  effectiveness  of  the  system. 


Figure  20.  Nine  step  methodology  to  characterize  MOEs  (from  G.  Eangford, 
Determinants  of  Deterrence  Effectiveness  2014,  12) 
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The  integrative  framework  was  formalized  using  four  of  the  possible  nine  cardinal 
points.  The  cardinal  points  in  the  process  frame’s  ‘models  and  representations’  and  the  object 
frame’s  ‘user  behaviors’  domain  were  addressed  but  MOEs  were  not  developed.  MOE 
development  and  analysis  within  these  domains’  cardinal  points  should  be  performed  once  a 
CMMS  is  established  as  a  program-of-record.  Once  the  CMMS  becomes  a  program-of- 
record  it  will  be  operated  under  a  set  of  standard  policies  and  procedures  that  will  govern  all 
users.  The  surface  maintenance  organizations  advancing  the  CBM+  implementation  are 
currently  in  the  CMMS  selection  and  implementation  stages.  The  CMMS  programs  currently 
being  used  are  operated  mostly  by  contractors;  government  maintenance  organization 
personnel  CMMS  access  is  usually  limited  to  data  receivers.  MOEs  related  to  user  behaviors 
towards  the  CMMS  and  its  functions  can  be  better  developed  once  the  maintenance 
organizations  establishes  a  specific  CMMS  as  the  organization’s  premier  management  tool 
and  the  users  become  familiar  with  it. 


Processes 

Co2nition 

Procedures 

Models  &  Rei>resenrnfion> 

Ohjcct.s 

User 

Belia\iors 

ConcepnuliZAtic^i  of  stakeholdet 
belmiors  (MOE  a)  when  the 
j>ro<hict*ser\nce  IS  used:  .md  when  it 
not  used  (or  available)  (MOE  p) 

Intluence  of  procedures  on  processes 
and  ineclimusnis  descnbing  user 
beliauors  due  to  product’seivice 
(MOE  i).  Influence  descnbuig  user 
beliaMoi^  due  to  lack  of 
pioduct*sei-\ice  (MOE-f> 

Comparison  of  expectations  of  models 
or  representations  of  stakeholder 
beluviors  to  actions  (MOE  t): 
Ewiliution  of  behaMors  to  predicted 
actions  (MOE  v) 

Fimctions 

Prognostication  of  consequences  of 
interactions  between  objects  tlirou$di 
exchange  of  EMMI  (MOE-r):  S: 
Expectations  of  interactions  (MOE-c) 

Availabilitj'  and  Validits'  of  processes 
and  inechanisiiis  that  detennine 
resource  utilizations  for  hinctions 
(MOE-ii):  Piocesses  and  ineclianisms 
that  dehne  the  boiuidary  conditions  for 
anticipated  operations  of  all  hmctions 
(MOE  b) 

Models  representations  show  ing  all 
huictional  perfomunces  (MOE-n). 
Models  representations  show  iug  all 
hmctional  perfonnance's  qiuilit\’ 

(MOE  q) 

Physical 
Entities  - 
Object 

Experience  with  posited  objects  (MOE- 
e)  &  anticipated  responses  of  posited 
objects  (MOE-r) 

A\  ailabilit>‘  and  \'alidit>’  of  processes 
and  ineclianisiiis  resulting  ui  the 
selection  and  de%  elopinent  of  all 
pliysical  elements  (MOE-s); 

Processes  and  luechaiusms  resultuig  in 
the  de\'elopinent  of  all  pln’sical 
elements  and  operational  contexts 
(MOE  X) 

Models  or  representations  of  all 
physical  elements,  tstnictures. 
properties,  traits,  and  atttibutes  (MOE- 
0):  Models  or  representations  of  all 
social,  political,  economic  elements 
(MOE  j) 

Eigure  21.  MOE  Eramework  Cardinal  Points  (from  G.  Eangford  Building  the 
Determinants  of  Technology  Effectiveness  2014,  12) 
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D.  FURTHER  RESEARCH 


The  scope  and  limitations  of  this  project  leave  opportunities  for  further  research 
concerning  MOE  development  and  CMMS  MOE  analysis. 

Eurther  research  should  develop  CMMS  MOEs  for  a  maintenance  organization  that 
uses  a  fully  employed  program-of-record  CMMS.  Once  the  CMMS  is  fully  employed  as  a 
program-of-record,  the  stakeholder  input  will  be  based  on  the  consequences  of  the 
established  policies  and  procedures  governing  the  system.  The  research  can  extend  CMMS 
MOE  development  to  the  nine  integrative  framework  cardinal  points  using  the  nine-step 
method  based  on  the  stakeholder’s  revised  needs  and  in-service  requirements  iterations. 

Although  the  nine-step  methodology  is  practicable  within  most  maintenance 
organizations  further  research  should  explore  other  repeatable  MOE  development 
alternatives. 

E.  CONCLUSION 

The  nine-step  method  provides  a  consistent  approach  that  may  be  used  as  a  tool  to 
develop  significant  MOEs  that  evaluate  the  successful  operational  use  of  a  CMMS  program 
in  the  implementation  of  a  CBM-i-  strategy  using  subjective  and  objective  criteria. 

Maintenance  organizations  should  adopt  a  consistent  methodical  approach,  such  as 
the  nine-step  method,  to  evaluate  the  fitness-for-purpose  of  their  maintenance  management 
systems.  The  consequence  of  not  using  a  methodical  approach  to  develop  MOEs  is  that  the 
maintenance  organization  might  waste  time,  money,  and  other  limited  resources  by 
measuring  inappropriate  program  characteristics  that  are  not  true  indicators  of  the  system’s 
success.  Additionally,  not  getting  a  true  measure  of  a  system’s  effectiveness  will  prolong  the 
inefficient  use  of  a  possibly  capable  system. 

Meaningful,  observable,  quantifiable,  and  precise  MOEs  provide  the  most  significant 
criteria  to  assess  the  quality  of  a  CMMS,  the  services  it  provides,  and  its  usage. 
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APPENDIX  A.  GLOSSARY 


adaptability 

“Attributes  of  software  that  bear  on  the  opportunity  for  its  adaptation  to  different 
specified  environments  without  applying  other  actions  or  means  than  those  provided 
for  this  purpose  for  the  software  considered”  (INCOSE  1998,  12). 

balanced  scorecard 

Strategic  planning  and  management  system  that  aligns  activities  with  the  overarching 
goals  of  an  organization  and  measures  performance  against  specific  goals.  (Balanced 
Scorecard  Institute  2014) 

boundary  conditions 

1 .  “Mediation  of  capabilities  that  enact  across  boundaries”  (G.O.  Langford  2012, 43). 

2.  Description  of  “the  conditions  which  determine  the  interaction  between  two 
objects”  (G.  O.  Langford  2012,  43). 

effectiveness 

1.  “The  extent  to  which  the  goals  of  the  system  are  attained”  (INCOSE  1998,  71). 

2.  “The  degree  to  which  a  system  can  be  elected  to  achieve  a  set  of  specific  mission 
requirements”  (INCOSE  1998,  71). 

flexibility 

Attribute  of  software  that  refers  to  its  ability  to  adapt  to  external  changes  (Wikipedia 
2013). 

function 

1.  “An  action/task  that  the  system  must  perform  to  satisfyy  customer  and  developer 
needs”  (INCOSE  1998,  98). 

2.  “The  action  or  actions,  which  an  item  is  desidned  to  perform”  (INCOSE  1998, 98). 
legacy  system 

“Systems  that  are  candidates  for  phase-out,  upgrade,  or  replacement”  (INCOSE 
1998,  128). 

maintenance 

1 .  “Those  actions  required  to  restore  or  maintain  an  item  to  a  serviceable  condition” 
(INCOSE  1998,  136). 

2.  “The  combination  of  all  technical  and  administrative  actions,  including 
supervision  actions,  intended  to  retain  an  item  in,  or  restore  it  to,  a  state  in  which  it 
can  perform  a  required  function”  (INCOSE  1998,  137). 

measure  of  effectiveness 
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“A  metric  used  to  quantify  the  performance  of  system  products  and  processes  in 
terms  that  describe  the  utility  or  value  when  executing  customer  missions”  (INCOSE 
1998,  144). 

mechanism 

“The  technical  aspects  of  doing  something”  (WordWeb  2014). 


object 

“Anything  physical,  comprised  of  matter  or  energy  in  ways  that  manifest  as  physical 
properties”  (G.  O.  Langford  2012,  44). 

operational  context 

The  external  environment  that  influences  an  object’s  operation  (Wikipedia  2013) 
organizational  level  maintenance 

“The  maintenance  and  repair  performed  by  the  activity  level  (organization)  which 
uses  the  system's  equipment,  within  the  activity's  capability”  (Cochrane  1995). 

performance  drivers 

“Critical  success  factors  that  determine  whether  business  and  marketing  objectives 
are  achieved”  (SensaCom  n.d.). 

power  user 

A  user  who  has  the  ability  to  use  advanced  features  of  programs  which  are  beyond 
the  abilities  of  average  users,  but  is  not  necessarily  capable  of  computer 
programming  and  system  administration  (Wikipedia  2013). 

program  of  record 

“Program  as  recorded  in  the  current  Future  Years  Defense  Program  (FYDP)  or  as 
updated  from  the  last  FYDP  by  approved  program”  (Defense  Acquisition  University 
2010). 

scalability 

“The  ability  to  use  the  same  application  software  on  many  different  classes  of 
hardware/software  platforms”  (INCOSE  1998,  210). 

suitableness 

“The  quality  of  having  the  properties  that  are  right  for  a  specific  purpose”  (WordWeb 
2014). 

Technical  Authority 

“The  authority,  responsibility,  and  accountability  to  establish,  monitor,  and  approve 
technical  standards,  tools  and  processes  in  conformance  to  higher  authority  policy, 
requirements,  architectures,  and  standards”  (Naval  Sea  Systems  Command  2008,  8- 
5). 
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validation 

Steps  taken  to  ensure  that  a  system  meets  the  requirements  (INCOSE  1998,  280) 


value  stream  mapping 

A  lean-manufaeturing  teehnique  that  analyzes  and  designs  flow  at  the  system  level 
with  the  objeetive  of  identifying  waste  in  the  eurrent  state  of  a  serviee  proeess. 
(Wikipedia  2013) 


waste 

1.  Any  non-required  step  or  aetion  in  a  proeess.  (GoLeanSixSigma.eom  n.d.) 


79 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


80 


APPENDIX  B.  CMMS  IMPLEMENTATION  SURVEY  RESULTS  -  2000 


These  are  the  results  from  a  CMMS  implementation  survey  conducted  in  2000  by  the 
Plant  Maintenance  Resource  Center.  (The  Plant  Maintenance  Resource  Center  2009) 

Overview 

This  survey  of  CMMS  Implementation  was  conducted  on  the  Plant  Maintenance 
Resource  Center  website  between  June  and  August  2000. 

Summary  of  Key  Findings 

Voluntary  (and  confidential)  responses  were  sought  to  the  survey,  and  87  valid 
responses  were  received  from  a  wide  range  of  individuals  working  across  a  variety  of 
industries. 

The  key  findings  are: 

Most  respondents  reported  some  or  significant  benefits  as  a  result  of  their  CMMS 
implementation,  however  a  significant  proportion  (between  20  and  40%)  of  respondents 
either  were  unable  to  identify  any  benefits,  or  reported  no  benefits. 

•  Overall,  a  significant  20%  of  respondents  rated  their  CMMS  implementation 
as  poor. 

•  Overall,  it  would  appear  that  conducting  BPR  seems  to  increase  the  chances 
of  successful  implementation-  although,  on  its  own,  still  not  increasing  the 
chance  of  success  above  50%. 

•  Statistical  analysis  of  responses  showed  no  clear  correlation  between  the 
conduct  of  other  usual  change  management  activities  associated  with  CMMS 
implementation,  and  the  perception  of  implementation  success 

•  However,  the  factors  stated  by  respondents  as  being  important  were  obtaining 
management  commitment,  selecting  the  correct  CMMS,  and  effective 
training. 

•  In  contrast,  those  currently  considering  replacing  or  upgrading  their  CMMS 
place  a  far  higher  emphasis  on  selecting  the  correct  CMMS,  and  do  not 
consider  training  to  be  an  important  implementation  issue. 

•  The  most  common  CMMS  in  use  by  respondents  were  Maximo,  MP2,  SAP 
and  MIMS. 


81 


•  Most  CMMS  were  seen  as  being  relatively  easy  to  use,  with  the  exception  of 
SAP.  SAP  was  also  considered  by  some,  to  be  functionally  poor. 

•  In  terms  of  the  perception  of  success,  Maximo  implementations  were 
generally  seen  as  successful,  while  SAP  users  were  divided  -  it  appears  that 
you  either  love  SAP  or  hate  it. 

Respondent  Data 

Of  the  87  valid  responses,  just  over  half  were  based  in  the  USA,  with  Australia, 
Canada,  and  the  United  Kingdom  also  well  represented. 


Country  | 

Responses 

%  of  Total 

United  States 

481 

55.2% 

Australia  | 

11 

United  Kingdom 

4 

4.6% 

Canada  | 

3 

3.4% 

New  Zealand  1 

.  .  1 

2 

2.3% 

South  Africa  | 

2 

2.3% 

Thailand  | 

2 

2.3% 

Belgium 

1 

1.1% 

Chile  1 

1 

1.1% 

Indonesia  | 

1 

1.1% 

Netherlands 

1 

1.1% 

Norway 

1 

1.1% 

Spain  1 

1 

1.1% 

Other/Not  Specified! 

9i 

Respondents  came  from  a  wide  range  of  industries. 


Industry 

Responses 

%  of  Total 

Manufacturing:  Other 

15 

1 7.2% 

Manufacturing-Metal  products 

10 

1 1 .5% 

Manufacturing-Food,  beverages,  tobacco 

7 

8.0% 

Utilities-Electricity  Generation 

8.0% 

Manufacturing-Petroleum  refining,  chemicals  and  associated  products 

5| 

5.7% 

Manufacturing-Wood  and  paper  products 

5| 

5.7% 

Oil  and  Gas-Oil  and  gas  extraction 

5 

5.7% 
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Industry 

Responses 

%  of  Total 

- 1 

Manufacturing-Machinery  and  equipment 

4 

Services-Heaithcare 

3 

3.4% 

Services-Property  services/Buiiding  Maintenance 

2 

2.3% 

Manufacturing-Non-metaiiic  minerai  processing 

1 

1.1% 

Mining-Metai  ore 

1 

1.1% 

Mining-Other 

1 

1.1% 

Services-Education/Academia 

1 

1.1% 

Services:  Research  &  Deveiopment 

1 

1.1% 

Services-Teiecommunications 

1 

1.1% 

Utiiities-Eiectricity  Transmission  and  Distribution 

1 

1.1% 

Utiiities-Water,  sewerage,  drainage 

1 

1.1% 

Other/Not  Specified 

15 

Maintenance  line  management  positions  and  Maintenance  Engineering  positions  were  well 
represented  in  the  survey. 


Position 

Responses 

%  of  Total 

Maintenance  Manager/Superintendent 

24 

27.6% 

Maintenance  Foreman/Supervisor 

12 

13.8% 

Maintenance  Pianner 

9 

10.3% 

Piant/Maintenance  Engineer 

9! 

10.3% 

Reiiabiiity  Engineer 

3| 

3.4% 

CEO/Managing  Director 

3| 

3.4% 

Maintenance  Contract  Manager 

2 

2.3% 

Product  Support  Manager 

2 

2.3% 

Software  Support  Manager 

2 

2.3% 

Software  Support  Professionai 

2 

2.3% 

Consuiting  Engineer 

2 

2.3% 

Maintenance  Crafts/Tradesperson 

1 

1.1% 

Maintenance  Technician 

1 

1.1% 

Process/Industriai  Engineer 

1 

1.1% 

Software  Support  Technician 

1 

1.1% 

Management  Consuitant 

1 

1.1% 

Other/Not  Specified 

12 

13.8% 
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Respondents  generally  came  from  larger  and  medium  sized  organizations 


No  of  T rades/Craftspeople 

Responses 

%of  Total 

Large(more  than  100  crafts/tradespeople) 

41 

47.1% 

Medium(1 0  to  1 00  crafts/tradespeople) 

30 

34.5% 

Small(less  than  10  crafts/tradespeople) 

10 

1 1 .5% 

Detailed  Results 

Full  statistical  survey  results  can  be  viewed  here.  This  article  focuses  on  some  of  the 
more  interesting  results,  and  also  reviews  the  correlation  between  some  of  the  survey  results 
in  order  to  determine  what  factors  (if  any)  are  more  likely  to  lead  to  successful  CMMS 
Implementation. 

CMMS  usage 

Almost  90%  of  respondents  are  currently  using  a  CMMS. 


Does  your  workplace  currently  use  a  CMMS? 

Responses 

%  of  Total 

Yes 

78 

89.7% 

No 

9 

10.3% 

The  most  common  CMMS  in  use  include  Maximo,  SAP,  MP2,  and  MIMS. 


Which  CMMS  does  your  workplace  currently  use? 

CMMS 

Responses' 

%of  Total 

Maximo 

16 

18.4% 

SAP 

12 

13.8% 

MP2 

11 

12.6% 

MIMS 

5 

5.7% 

Mainpac 

3 

3.4% 

Mainsaver 

2 

2.3% 

MS2000 

2 

2.3% 

PMC 

2 

2.3% 

Tabware 

2 

2.3% 
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Which  CMMS  does  your  workplace  currently  use? 

CMMS  1  Responses  1%  of  Total 

AMMS 

1 

1.1% 

Benchmate 

1 

1.1% 

Developed  in-house 

1 

1.1% 

Elke 

1 

1.1% 

Frontline 

1 

1.1% 

GPMate 

1 

1.1% 

IMMPOWER 

1 

1.1% 

Impactxp 

1 

1.1% 

Insight  Plant  Maintenance  System 

1 

1.1% 

Mainstar 

1 

1.1% 

Maintain  it 

1 

1.1% 

Maintenance  Manager 

1 

1.1% 

Maintenance  Tracker 

1 

1.1% 

MAPCON 

1 

1.1% 

Marcam  PRISM 

1 

1.1% 

MCP 

1 

1.1% 

Mex 

1 

1.1% 

OCS  Materials  Management 

1 

1.1% 

Passport 

1 

1.1% 

pcmaint32 

1 

1.1% 

PERMAC 

1 

1.1% 

PM  Plus 

1 

1.1% 

So  max 

1 

1.1% 

Ultimo 

1 

1.1% 

Other/Not  Specified 

8 

9.2% 
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Most  systems  have  been  in  plaee  only  for  a  few  years. 


How  long  ago  did  you  ‘ 

‘go  live”? 

Years  | 

Responses 

%  of  Total 

In  progress! 

9i 

10.3% 

<1  year  j 

12 

13.8% 

1  -2  years  j 

20 

23.0% 

2-3  years  j 

9| 

10.3% 

5 

5.7% 

>4-5  years  j 

10 

1 1 .5% 

>5  years  j 

14 

16.1% 

Reasons  for  selecting  and  implementing  a  CMMS 

Many  CMMS  Implementations  were  eondueted  with  the  expeetation  that  they  would 
lead  to  improved  maintenanee  performanee.  However,  there  were  also  many  other  reasons 
for  implementing  a  CMMS. 


What  was  the  main  reason  that  your  workplace  changed  CMMS,  or  implemented  a  new 

CMMS? 

Reason 

Responses 

%  of  Total 

To  improve  maintenance  performance 

25 

28.7% 

Improved  functionality  and  features 

16 

18.4% 

To  integrate  the  Maintenance  system  with  other  systems 

9i 

10.3% 

Don't  know 

8 

9.2% 

Year  2000  compliance  problems 

7| 

8.0% 

Vendor  no  longer  supported  our  old  CMMS 

2 

2.3% 

To  comply  with  company  standards 

2 

2.3% 

To  use  newer  technology 

1 

1.1% 

Other 

7 
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A  large  proportion  of  respondents  were  not  aware  of  the  reasons  for  seleetion  of  the 
eurrent  CMMS. 


What  was  the  most/second  most  important  reason  that  your  workplace  chose  your  current 

CMMS? 


Most  Important 

Second  Most  Important 

Reason 

Responses 

%  of  Total 

Responses 

%  of  Total 

Availability  of  local  support  | 

2| 

2.3% 

1 

1.1% 

Compatibility  with  previous  CMMS 
software  | 

2 

2.3% 

1 

1.1% 

Don't  know  | 

20 

23.0% 

23 

26.4% 

Ease  of  implementation 

1 

1.1% 

5 

5.7% 

Ease  of  Use 

11 

8 

9.2% 

General  functionality  and  features 

16 

18.4% 

15 

1 7.2% 

Integration  with  other  commercial  software  | 

9 

10.3% 

5 

5.7% 

Other  1 

9 

10.3% 

4 

4.6% 

Price 

7 

6 

6.9% 

Speed  of  system  response 

1 

1.1% 

1 

1.1% 

CMMS  Comparison 

Overall,  CMMS  are  seen  as  being  moderately  easy  to  use,  with  Maximo  rating 
highly,  and  SAP  seen  as  being  harder  to  use. 


How  would  you  rate  your  current  CMMS  in  terms  of  its  ease  of  use? 

No  of  Responses 


CMMS 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor  1 

Maximo 

51 

5 

3 

3 

MP2 

1 

3 

4 

2 

1 

SAP 

4 

2 

1 

5 

MIMS 

1 

2 

1 

1 

4l 

8i 

14 

6 

3 

Total  1 

1 

lOi 

21 

25 

13 

101 
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Most  CMMS  are  seen  as  being  well  endowed  with  functionality  and  features, 
although  again,  SAP  rates  poorly  according  to  some. 


How  would  you  rate  your  current  CMMS  in  terms  of  its  general  features  and  functionality? 

No  of  Responses 


CMMS 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor  1 

Maximo 

5 

5 

3 

1 

1 

4 

2 

1 

SAP 

2 

3 

6 

4 

1 

5 

9 

3 

6 

Total 

13 

25 

9 

14 

Maximo  implementations  are  generally  seen  as  successful,  while  SAP  users  either 
love  it,  or  hate  it.  Overall,  a  significant  20%  of  respondents  rated  their  implementation  as 
poor. 


Overall,  how  would  you  rate  the  success  of  your  CMMS  implementation? 

No  of  Responses 


CMMS 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor 

Maximo 

1 

8| 

3 

2 

2 

MP2 

1 

2 

4 

2 

2 

3I 

3 

1 

5 

3 

2 

2 

7 

lOi 

9l 

6 

7l 

20 

20 

14 

17 

Factors  Influencing  Implementation  Success 

If  you  look  at  the  results  for  the  question  of  Business  Process  Reengineering  (BPR) 
impact  on  implementation  success  you  see  that  of  respondents  that  had  conducted  BPR  16 
rated  their  success  as  Good  to  Excellent  and  4  rated  their  success  as  Satisfactory  to  Poor.  (4 
to  1  ratio).  If  you  compare  that  to  respondents  that  did  not  conduct  BPR  where  25  rated  Good 
to  Excellent  and  20  rated  Satisfactory  to  Poor  (5-4  ratio).  This  is  a  significant  indicator  that 
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there  was  a  larger  percentage  of  dissatisfaction  if  BPR  was  not  conducted,  and  so  it  could  be 
said  that  BPR  is  a  significant  factor  contributing  to  implementation  success.  On  the  other 
hand,  if  you  look  at  the  proportion  of  those  who  rated  their  implementation  as  Excellent  or 
Very  Good,  compared  with  those  who  considered  it  Satisfactory  or  Poor,  in  both  cases,  the 
majority  of  respondents  had  not  conducted  BPR  (13  of  23,  ignoring  the  Don't  Knows,  and  20 
of  24,  respectively).  This  indicates  that  conducting  BPR  was  not  a  significant  factor  in 
assisting  with  implementation  success.  Overall,  however,  it  would  appear  that  conducting 
BPR  does  seem  to  increase  the  chances  of  success  -  although,  on  its  own,  still  not  increasing 
the  chance  of  success  above  50%. 


Impact  of  Business  Process  Reengineering  on  Implementation  Success 

How  do  you  rate  success  of  Implementation? 

No  of  Responses 


BPR  Conducted? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor  I 

Yes 

4 

6 

6 

2 

2| 

No 

2 

11 

12 

11 

Don't  Know 

1 

3 

2 

l| 

_5j 

Total 

7 

20 

20 

14l 

Using  consultants  to  assist  with  implementation  also  appears  to  have  minimal 
influence  on  the  perception  of  implementation  success. 


Impact  of  Consultants  on  Implementation  Success 

How  do  you  rate  success  of  Implementation? 
No  of  Responses 


fused  Consultants? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor 

Yes 

3 

- 1 

9 

5l 

5 

7 

No 

4 

io! 

14 

8 

m 

Don't  Know 

1 

1 

11 

m 

Total 

7 

20 

20 

14 

17 
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Issuing  regular  newsletters  during  implementation  also  appears  to  have  minimal 
influence  on  the  perception  of  implementation  success. 


Impact  of  Newsletters  on  Implementation  Success 

How  do  you  rate  success  of  Implementation? 
No  of  Responses 


Issued  Newsletters? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poorj 

Yes 

41 

2 

5l 

3 

8 

No 

2 

14 

15 

11 

■! 

Don't  Know 

1 

3 

Total 

7l 

19 

20 

14 

16 

Performing  regular  face-to-face  briefings  during  implementation  appears  to  have  a 
minor  influence  on  the  perception  of  implementation  success. 


Impact  of  Face  to  Face  Briefings  on  Implementation  Success 

How  do  you  rate  success  of  Implementation? 
No  of  Responses 


(Regular  Briefings? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor 

Yes 

5i 

13 

12 

9 

6 

No 

1 

51 

8| 

4 

m 

Don't  Know 

1 

2 

i; 

m 

Total 

7 

20 

20 

14i 

17 

Establishing  a  help  line  during  implementation  appears  to  have  a  minor  influence  on 
the  perception  of  implementation  success. 


Impact  of  a  “Help  Line”  on  Implementation  Success 


How  do  you  rate  success  of  Implementation? 
No  of  Responses 


Help  Line? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor 

Yes 

41 

loi 

12 

6 

9 

No 

2 

6 

10 

8 

■1 

1 

2 

20 


20 


Total 


90 


14 


16 


Conducting  regular  stakeholder  analysis  during  implementation,  although  not  often 
done,  appears  to  have  a  positive  influence  on  the  perception  of  implementation  success. 


Impact  of  Stakeholder  Analysis  on  Implementation  Success 


How  do  you  rate  success  of  Implementation? 
No  of  Responses 


Stakeholder  Analysis? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor 

Yes 

3 

6 

4 

1 

2 

No 

3 

12 

15 

8 

Don't  Know 

1 

2 

1 

2 

ei 

Total 

20 

20 

131 

16 

Conducting  Training  Needs  analysis  as  part  of  implementation  appears  to  have  a 
minor  influence  on  the  perception  of  implementation  success. 


Impact  of  Training  Needs  Analysis  on  Implementation  Success 


How  do  you  rate  success  of  Implementation? 
No  of  Responses 

Training  Needs  Analysis? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor 

Yes 

5 

9 

11 

6i 

m 

1 

No 

1 

9! 

9 

7 

7 

Don't  Know 

1 

1 

1 

5 

Total 

7 

19 

20 

14 

16 

Conducting  Workforce  Culture  Surveys  as  part  of  implementation  appears  to  have  no 
influence  on  the  perception  of  implementation  success. 


Impact  of  Culture  Surveys  on  Implementation  Success 


How  do  you  rate  success  of  Implementation? 
No  of  Responses 


1  Culture  Surveys? 

Excellent 

Very  Good 

Good 

Satisfactory 

Poor| 

Yes 

5 

4 

5 

3 

2 

No 

1 

14 

14 

8 

. 

11 

Don't  Know 

1 

1 

1 

3 

3i 

Total 

7 

_ 

20 

14l 

16 

According  to  the  respondents,  the  most  important  factors  in  their  success  were 
obtaining  Senior  Management  commitment,  and  selecting  the  correct  CMMS. 


What  do  you  consider  are  the  two  most  important  aspects  of  your  impiementation  that  ied 

to  your  success? 


Responses 

Factor 

Most  important 

Second  Most  important 

1  Totaij 

Senior  Management  commitment 

11 

13 

24 

Choosing  the  right  CMMS 

13 

6 

19 

Effective  training 

5 

8 

13 

Focus  on  business  benefits 

8 

5 

13 

Adequate  budget 

3 

7 

10 

Effective  Change  Management 

6 

4 

10 

Effective  Project  Management 

5 

5 

10 

CMMS  Vendor  Support 

5 

2 

7 

Effective  BPR 

3 

4 

7 

Consuitant  support 

1 

4 

Other 

13 

7 

Furthermore,  the  most  important  area  in  which  respondents  wished  they  had  done 
better  was  in  the  area  of  training. 


in  hindsight,  what  is  the  most  important  aspect  of  your  impiementation  that  you  shouid 
have  spent  more  time  and  effort  on,  in  order  to  increase  impiementation  success? 


Factor 

Responses 

Percent 

Effective  training 

21 

24.1% 

Choosing  the  right  CMMS 

17 

19.5% 

Senior  Management  commitment 

8 

9.2%j 

Effective  BPR 

8 

9.2% 

Effective  Change  Management 

6 

6.9% 

Effective  Project  Management 

5 

5.7% 

Adequate  budget 

3 

3.4% 

Focus  on  business  benefits 

3 

3.4% 

CMMS  Vendor  Support 

1 

1.1% 

Other 

2 

2.3% 

In  comparison,  those  currently  considering  implementing  a  new  CMMS  place  far 
greater  importance  on  CMMS  selection.  Effective  Training  is  not  considered  to  be  at  all 
important. 


What  do  you  consider  will  be  the  most  important  aspect  of  your  implementation  that  will 

lead  to  success? 


Responses 

Factor 

Most  Important 

Second  Most  Important 

Total 

Choosing  the  right  CMMS 

9i 

3 

Senior  Management  commitment 

1 

2 

3i 

5 

Effective  Change  Management 

2 

3 

5 

Effective  Project  Management 

2 

1 

3 

Benefits  obtained  from  CMMS  Implementation 

Overall,  most  respondents  reported  that  their  CMMS  implementation  has  led  to  some 
or  significant  benefits.  However  a  large  proportion  (between  20%  and  40%)  of  respondents 
either  reported  achieving  no  business  benefits  from  their  CMMS  implementations,  or  were 
unable  to  quantify  benefits. 


Size  of  Benefits  Obtained 
%  of  Responses 


Area  of  Benefit 

Significant 

Some 

None 

Don't  Know| 

Reductions  in  Labor  Costs 

1 1 .5% 

Reductions  in  Materiais  Costs 

1 1 .5% 

Reductions  in  Other  Costs 

13.8% 

improved  Equipment  Avaiiabiiity 

33.3% 

9.2% 

improved  Equipment  Reiiabiiity 

21.8% 

35.6% 

24.1% 

improved  Cost  Controi 

44.8% 

2.3% 

improved  Maintenance  History 

23.0% 

2.3% 

improved  Maintenance  Pianning 

32.2% 

18.4% 

2.3% 

improved  Maintenance  Scheduiing 

31.0% 

18.4% 

2.3% 

improved  Maintenance  Scheduies 

2.3% 

improved  Spare  Parts  Controi 

24.1% 

4.6% 
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Amazingly,  only  20%  of  organizations  responding  have  attempted  to  formally 
quantify  the  benefits  obtained  from  their  CMMS  implementation. 


1  Has  your  workplace  formally  measured  the  benefits  of  your  CMMS  implementation? 

1 

Responses 

%  of  Totai 

Yes 

17 

19.5% 

No 

51 

58.6% 

Don't  Know  | 

9i 

10.3% 
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